[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708447806542-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 19:00:47 dist-test-slave-lqkv krb5kdc[399](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 19:00:47 dist-test-slave-lqkv krb5kdc[399](info): set up 2 sockets
Sep 29 19:00:47 dist-test-slave-lqkv krb5kdc[399](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260929 19:00:48.832989 392 mini_kdc.cc:121] Time spent starting KDC: real 1.009s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708447806542-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708447806542-392-0/krb5kdc/test-user.keytab.
Sep 29 19:00:48 dist-test-slave-lqkv krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708448, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T19:00:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T19:00:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708447806542-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708447806542-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 19:00:48.939481 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUbnLHE/build/release/bin/kudu
/tmp/dist-test-taskUbnLHE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708447806542-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708447806542-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708447806542-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708447806542-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36981
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708447806542-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36981
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708447806542-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708447806542-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708447806542-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:00:49.016412 412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:00:49.016600 412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:00:49.016621 412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:00:49.017695 412 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 19:00:49.017731 412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:00:49.017745 412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:00:49.017758 412 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 19:00:49.017769 412 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 19:00:49.019114 412 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708447806542-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708447806542-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36981
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708447806542-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36981
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708447806542-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708447806542-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 18:09:13 UTC on ccycloud.quasar-hpkalf.root.comops.site
build id 4680913
I20260929 19:00:49.019282 412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:00:49.019447 412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 19:00:49.024488 412 server_base.cc:1034] running on GCE node
W20260929 19:00:49.024663 420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:00:49.024647 418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:00:49.024631 417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:00:49.025097 412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:00:49.025389 412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:00:49.026564 412 hybrid_clock.cc:648] HybridClock initialized: now 1790708449026540 us; error 37 us; skew 500 ppm
Sep 29 19:00:49 dist-test-slave-lqkv krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708449, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:00:49.033017 412 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 19:00:49.033452 412 webserver.cc:460] Webserver started at http://127.0.98.62:42273/ using document root <none> and password file <none>
I20260929 19:00:49.033643 412 fs_manager.cc:359] Metadata directory not provided
I20260929 19:00:49.033682 412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:00:49.033766 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:00:49.034744 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708447806542-392-0/minicluster-data/master-0/data/instance:
uuid: "e0e55d9c98f847c0915f1d8a8f205253"
format_stamp: "Formatted at 2026-09-29 19:00:49 on dist-test-slave-lqkv"
server_key: "37ccb21bd5c68e9edd7745f8a2c01482"
server_key_iv: "f3fb0824c7fd584a210a4f5a8f66dcb9"
server_key_version: "encryptionkey@0"
I20260929 19:00:49.035044 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708447806542-392-0/minicluster-data/master-0/wal/instance:
uuid: "e0e55d9c98f847c0915f1d8a8f205253"
format_stamp: "Formatted at 2026-09-29 19:00:49 on dist-test-slave-lqkv"
server_key: "37ccb21bd5c68e9edd7745f8a2c01482"
server_key_iv: "f3fb0824c7fd584a210a4f5a8f66dcb9"
server_key_version: "encryptionkey@0"
I20260929 19:00:49.036276 412 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 19:00:49.037014 427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:00:49.037209 412 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 19:00:49.037294 412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708447806542-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708447806542-392-0/minicluster-data/master-0/wal
uuid: "e0e55d9c98f847c0915f1d8a8f205253"
format_stamp: "Formatted at 2026-09-29 19:00:49 on dist-test-slave-lqkv"
server_key: "37ccb21bd5c68e9edd7745f8a2c01482"
server_key_iv: "f3fb0824c7fd584a210a4f5a8f66dcb9"
server_key_version: "encryptionkey@0"
I20260929 19:00:49.037376 412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708447806542-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708447806542-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708447806542-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:00:49.062216 412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:00:49.064162 412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:00:49.064273 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:00:49.068282 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36981
I20260929 19:00:49.068339 479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36981 every 8 connection(s)
I20260929 19:00:49.068667 412 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708447806542-392-0/minicluster-data/master-0/data/info.pb
I20260929 19:00:49.069550 480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:00:49.071707 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0e55d9c98f847c0915f1d8a8f205253: Bootstrap starting.
I20260929 19:00:49.072257 480 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e0e55d9c98f847c0915f1d8a8f205253: Neither blocks nor log segments found. Creating new log.
I20260929 19:00:49.072561 480 log.cc:824] T 00000000000000000000000000000000 P e0e55d9c98f847c0915f1d8a8f205253: Log is configured to *not* fsync() on all Append() calls
I20260929 19:00:49.073261 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0e55d9c98f847c0915f1d8a8f205253: No bootstrap required, opened a new log
I20260929 19:00:49.074184 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUbnLHE/build/release/bin/kudu as pid 412
I20260929 19:00:49.074292 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708447806542-392-0/minicluster-data/master-0/wal/instance
I20260929 19:00:49.075042 392 external_mini_cluster.cc:1428] Setting key 1de69831ffeca4b4f75d6fd288ea3ea8
I20260929 19:00:49.075423 480 raft_consensus.cc:348] T 00000000000000000000000000000000 P e0e55d9c98f847c0915f1d8a8f205253 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0e55d9c98f847c0915f1d8a8f205253" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36981 } }
I20260929 19:00:49.075598 480 raft_consensus.cc:374] T 00000000000000000000000000000000 P e0e55d9c98f847c0915f1d8a8f205253 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:00:49.075625 480 raft_consensus.cc:729] T 00000000000000000000000000000000 P e0e55d9c98f847c0915f1d8a8f205253 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0e55d9c98f847c0915f1d8a8f205253, State: Initialized, Role: FOLLOWER
I20260929 19:00:49.075809 480 consensus_queue.cc:260] T 00000000000000000000000000000000 P e0e55d9c98f847c0915f1d8a8f205253 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0e55d9c98f847c0915f1d8a8f205253" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36981 } }
I20260929 19:00:49.075889 480 raft_consensus.cc:388] T 00000000000000000000000000000000 P e0e55d9c98f847c0915f1d8a8f205253 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 19:00:49.075927 480 raft_consensus.cc:482] T 00000000000000000000000000000000 P e0e55d9c98f847c0915f1d8a8f205253 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 19:00:49.075958 480 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e0e55d9c98f847c0915f1d8a8f205253 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:00:49.077018 480 raft_consensus.cc:504] T 00000000000000000000000000000000 P e0e55d9c98f847c0915f1d8a8f205253 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0e55d9c98f847c0915f1d8a8f205253" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36981 } }
I20260929 19:00:49.077122 480 leader_election.cc:302] T 00000000000000000000000000000000 P e0e55d9c98f847c0915f1d8a8f205253 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e0e55d9c98f847c0915f1d8a8f205253; no voters:
I20260929 19:00:49.077272 480 leader_election.cc:288] T 00000000000000000000000000000000 P e0e55d9c98f847c0915f1d8a8f205253 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 19:00:49.077320 485 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e0e55d9c98f847c0915f1d8a8f205253 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:00:49.077493 485 raft_consensus.cc:686] T 00000000000000000000000000000000 P e0e55d9c98f847c0915f1d8a8f205253 [term 1 LEADER]: Becoming Leader. State: Replica: e0e55d9c98f847c0915f1d8a8f205253, State: Running, Role: LEADER
I20260929 19:00:49.077517 480 sys_catalog.cc:564] T 00000000000000000000000000000000 P e0e55d9c98f847c0915f1d8a8f205253 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 19:00:49.077590 485 consensus_queue.cc:237] T 00000000000000000000000000000000 P e0e55d9c98f847c0915f1d8a8f205253 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0e55d9c98f847c0915f1d8a8f205253" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36981 } }
I20260929 19:00:49.077901 485 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0e55d9c98f847c0915f1d8a8f205253 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e0e55d9c98f847c0915f1d8a8f205253. Latest consensus state: current_term: 1 leader_uuid: "e0e55d9c98f847c0915f1d8a8f205253" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0e55d9c98f847c0915f1d8a8f205253" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36981 } } }
I20260929 19:00:49.077914 487 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0e55d9c98f847c0915f1d8a8f205253 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e0e55d9c98f847c0915f1d8a8f205253" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0e55d9c98f847c0915f1d8a8f205253" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36981 } } }
I20260929 19:00:49.077991 485 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0e55d9c98f847c0915f1d8a8f205253 [sys.catalog]: This master's current role is: LEADER
I20260929 19:00:49.077996 487 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0e55d9c98f847c0915f1d8a8f205253 [sys.catalog]: This master's current role is: LEADER
I20260929 19:00:49.078234 491 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 19:00:49.078783 491 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 19:00:49.080412 491 catalog_manager.cc:1347] Generated new cluster ID: d28b2ca9fa054cf7b24e04a677600d79
I20260929 19:00:49.080464 491 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 29 19:00:49 dist-test-slave-lqkv krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708448, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 19:00:49.092955 483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:49.075957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55848 (local address 127.0.98.62:36981)
0929 19:00:49.076154 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0929 19:00:49.076162 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 19:00:49.076490 (+ 328us) server_negotiation.cc:407] Connection header received
0929 19:00:49.078983 (+ 2493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:00:49.078991 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:00:49.079053 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:00:49.079167 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:00:49.081347 (+ 2180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:49.082699 (+ 1352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:49.084185 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:49.084763 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:49.084875 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:49.088990 (+ 4115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:00:49.088994 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:00:49.088994 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:00:49.089004 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:00:49.090972 (+ 1968us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:00:49.091420 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:00:49.091423 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:00:49.091423 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:00:49.091531 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:00:49.091795 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:00:49.091796 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:00:49.091796 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:00:49.092094 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:00:49.092249 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0929 19:00:49.092531 (+ 282us) server_negotiation.cc:299] Negotiation successful
0929 19:00:49.092668 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":97,"threads_started":1}
I20260929 19:00:49.093611 491 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 19:00:49.094067 491 catalog_manager.cc:1504] Loading token signing keys...
I20260929 19:00:49.098870 491 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e0e55d9c98f847c0915f1d8a8f205253: Generated new TSK 0
I20260929 19:00:49.099015 491 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708447806542-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708447806542-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260929 19:00:49.126884 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUbnLHE/build/release/bin/kudu
/tmp/dist-test-taskUbnLHE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708447806542-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708447806542-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708447806542-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708447806542-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36981
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708447806542-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708447806542-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708447806542-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708447806542-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:00:49.212044 506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:00:49.212225 506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:00:49.212247 506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:00:49.213413 506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:00:49.213452 506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:00:49.213479 506 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 19:00:49.214850 506 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708447806542-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708447806542-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708447806542-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708447806542-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36981
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708447806542-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 18:09:13 UTC on ccycloud.quasar-hpkalf.root.comops.site
build id 4680913
I20260929 19:00:49.215066 506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:00:49.215297 506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:00:49.220745 511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:00:49.220753 512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:00:49.220918 514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:00:49.221004 506 server_base.cc:1034] running on GCE node
I20260929 19:00:49.221249 506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:00:49.221498 506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:00:49.222652 506 hybrid_clock.cc:648] HybridClock initialized: now 1790708449222639 us; error 30 us; skew 500 ppm
Sep 29 19:00:49 dist-test-slave-lqkv krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708449, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:00:49.225421 506 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 19:00:49.225854 506 webserver.cc:460] Webserver started at http://127.0.98.1:38157/ using document root <none> and password file <none>
I20260929 19:00:49.226080 506 fs_manager.cc:359] Metadata directory not provided
I20260929 19:00:49.226145 506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:00:49.226289 506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:00:49.227262 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708447806542-392-0/minicluster-data/ts-0/data/instance:
uuid: "a91d2934d14e4786a90e209b7700603d"
format_stamp: "Formatted at 2026-09-29 19:00:49 on dist-test-slave-lqkv"
server_key: "b492e680e3ec23af22cc8386cc3cbcad"
server_key_iv: "94b110489674e0006583782b8cc95211"
server_key_version: "encryptionkey@0"
I20260929 19:00:49.227607 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708447806542-392-0/minicluster-data/ts-0/wal/instance:
uuid: "a91d2934d14e4786a90e209b7700603d"
format_stamp: "Formatted at 2026-09-29 19:00:49 on dist-test-slave-lqkv"
server_key: "b492e680e3ec23af22cc8386cc3cbcad"
server_key_iv: "94b110489674e0006583782b8cc95211"
server_key_version: "encryptionkey@0"
I20260929 19:00:49.228909 506 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260929 19:00:49.229580 521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:00:49.229737 506 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 19:00:49.229828 506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708447806542-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708447806542-392-0/minicluster-data/ts-0/wal
uuid: "a91d2934d14e4786a90e209b7700603d"
format_stamp: "Formatted at 2026-09-29 19:00:49 on dist-test-slave-lqkv"
server_key: "b492e680e3ec23af22cc8386cc3cbcad"
server_key_iv: "94b110489674e0006583782b8cc95211"
server_key_version: "encryptionkey@0"
I20260929 19:00:49.229916 506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708447806542-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708447806542-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708447806542-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:00:49.237668 506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:00:49.239306 506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:00:49.239454 506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:00:49.239682 506 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:00:49.239992 506 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:00:49.240046 506 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:00:49.240085 506 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:00:49.240110 506 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:00:49.246344 506 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37773
I20260929 19:00:49.246424 634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37773 every 8 connection(s)
I20260929 19:00:49.246686 506 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708447806542-392-0/minicluster-data/ts-0/data/info.pb
I20260929 19:00:49.251543 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUbnLHE/build/release/bin/kudu as pid 506
I20260929 19:00:49.251613 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708447806542-392-0/minicluster-data/ts-0/wal/instance
I20260929 19:00:49.251773 392 external_mini_cluster.cc:1428] Setting key 9eb8ccaac9c6098508e6a9ace6169687
Sep 29 19:00:49 dist-test-slave-lqkv krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708449, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 19:00:49.255664 637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:49.247360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36981 (local address 127.0.98.1:38137)
0929 19:00:49.247508 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0929 19:00:49.247537 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0929 19:00:49.247942 (+ 405us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:00:49.248151 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:00:49.248153 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:00:49.248678 (+ 525us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:00:49.248920 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:00:49.248925 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:00:49.249568 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:00:49.249569 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:00:49.250322 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:00:49.250323 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:00:49.250658 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:00:49.250658 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:00:49.250750 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:49.251171 (+ 421us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:00:49.251178 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:00:49.252622 (+ 1444us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:00:49.254270 (+ 1648us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:00:49.254275 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:00:49.254277 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:00:49.254587 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:00:49.254821 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:00:49.254822 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:00:49.254822 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:00:49.254956 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:00:49.255230 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:00:49.255234 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:00:49.255333 (+ 99us) client_negotiation.cc:769] Sending connection context
0929 19:00:49.255422 (+ 89us) client_negotiation.cc:240] Negotiation successful
0929 19:00:49.255527 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":92,"threads_started":1}
I20260929 19:00:49.255692 483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:49.247320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38137 (local address 127.0.98.62:36981)
0929 19:00:49.247365 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0929 19:00:49.247367 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:00:49.247628 (+ 261us) server_negotiation.cc:407] Connection header received
0929 19:00:49.248026 (+ 398us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:00:49.248027 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:00:49.248037 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:00:49.248088 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:00:49.249004 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:49.249480 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:49.250398 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:49.250585 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:49.250643 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:49.252712 (+ 2069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:00:49.252713 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:00:49.252713 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:00:49.252722 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:00:49.254193 (+ 1471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:00:49.254669 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:00:49.254670 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:00:49.254671 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:00:49.254750 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:00:49.255020 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:00:49.255022 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:00:49.255022 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:00:49.255166 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:00:49.255236 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 19:00:49.255467 (+ 231us) server_negotiation.cc:299] Negotiation successful
0929 19:00:49.255553 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260929 19:00:49.256876 635 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36981
I20260929 19:00:49.256966 635 heartbeater.cc:464] Registering TS with master...
I20260929 19:00:49.257123 635 heartbeater.cc:511] Master 127.0.98.62:36981 requested a full tablet report, sending...
I20260929 19:00:49.257740 444 ts_manager.cc:194] Registered new tserver with Master: a91d2934d14e4786a90e209b7700603d (127.0.98.1:37773)
I20260929 19:00:49.258628 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:38137
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708447806542-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708447806542-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260929 19:00:49.273826 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUbnLHE/build/release/bin/kudu
/tmp/dist-test-taskUbnLHE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708447806542-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708447806542-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708447806542-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708447806542-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36981
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708447806542-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708447806542-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708447806542-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708447806542-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:00:49.352116 640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:00:49.352301 640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:00:49.352324 640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:00:49.353487 640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:00:49.353526 640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:00:49.353552 640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 19:00:49.354918 640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708447806542-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708447806542-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708447806542-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708447806542-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36981
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708447806542-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 18:09:13 UTC on ccycloud.quasar-hpkalf.root.comops.site
build id 4680913
I20260929 19:00:49.355124 640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:00:49.355350 640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:00:49.360039 646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:00:49.360055 640 server_base.cc:1034] running on GCE node
W20260929 19:00:49.360039 648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:00:49.360046 645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:00:49.360450 640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:00:49.360677 640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:00:49.361807 640 hybrid_clock.cc:648] HybridClock initialized: now 1790708449361799 us; error 26 us; skew 500 ppm
Sep 29 19:00:49 dist-test-slave-lqkv krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708449, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:00:49.364050 640 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 19:00:49.364475 640 webserver.cc:460] Webserver started at http://127.0.98.2:33927/ using document root <none> and password file <none>
I20260929 19:00:49.364684 640 fs_manager.cc:359] Metadata directory not provided
I20260929 19:00:49.364727 640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:00:49.364883 640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:00:49.365850 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708447806542-392-0/minicluster-data/ts-1/data/instance:
uuid: "e3e00192fbf74d859664323411380e39"
format_stamp: "Formatted at 2026-09-29 19:00:49 on dist-test-slave-lqkv"
server_key: "3f65a631d0832a44fcb6413fb97b9d93"
server_key_iv: "0c56222e5389bc01de0bd7e2bd371d9c"
server_key_version: "encryptionkey@0"
I20260929 19:00:49.366194 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708447806542-392-0/minicluster-data/ts-1/wal/instance:
uuid: "e3e00192fbf74d859664323411380e39"
format_stamp: "Formatted at 2026-09-29 19:00:49 on dist-test-slave-lqkv"
server_key: "3f65a631d0832a44fcb6413fb97b9d93"
server_key_iv: "0c56222e5389bc01de0bd7e2bd371d9c"
server_key_version: "encryptionkey@0"
I20260929 19:00:49.367846 640 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 19:00:49.368582 655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:00:49.368755 640 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 19:00:49.368871 640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708447806542-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708447806542-392-0/minicluster-data/ts-1/wal
uuid: "e3e00192fbf74d859664323411380e39"
format_stamp: "Formatted at 2026-09-29 19:00:49 on dist-test-slave-lqkv"
server_key: "3f65a631d0832a44fcb6413fb97b9d93"
server_key_iv: "0c56222e5389bc01de0bd7e2bd371d9c"
server_key_version: "encryptionkey@0"
I20260929 19:00:49.368952 640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708447806542-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708447806542-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708447806542-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:00:49.378834 640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:00:49.380731 640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:00:49.380901 640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:00:49.381108 640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:00:49.381397 640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:00:49.381449 640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:00:49.381488 640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:00:49.381515 640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:00:49.387749 640 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41775
I20260929 19:00:49.387789 768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41775 every 8 connection(s)
I20260929 19:00:49.388088 640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708447806542-392-0/minicluster-data/ts-1/data/info.pb
Sep 29 19:00:49 dist-test-slave-lqkv krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708449, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 19:00:49.397122 483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:49.388772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48379 (local address 127.0.98.62:36981)
0929 19:00:49.388846 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0929 19:00:49.388849 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 19:00:49.389415 (+ 566us) server_negotiation.cc:407] Connection header received
0929 19:00:49.389842 (+ 427us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:00:49.389843 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:00:49.389853 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:00:49.389901 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:00:49.390759 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:49.391241 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:49.392171 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:49.392381 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:49.392438 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:49.394481 (+ 2043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:00:49.394482 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:00:49.394482 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:00:49.394488 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:00:49.395713 (+ 1225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:00:49.396100 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:00:49.396101 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:00:49.396102 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:00:49.396159 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:00:49.396445 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:00:49.396446 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:00:49.396446 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:00:49.396549 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:00:49.396603 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0929 19:00:49.396998 (+ 395us) server_negotiation.cc:299] Negotiation successful
0929 19:00:49.397062 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 19:00:49.397151 771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:49.388736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36981 (local address 127.0.98.2:48379)
0929 19:00:49.388929 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0929 19:00:49.388962 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0929 19:00:49.389762 (+ 800us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:00:49.389960 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:00:49.389963 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:00:49.390499 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:00:49.390675 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:00:49.390679 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:00:49.391323 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:00:49.391324 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:00:49.392087 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:00:49.392088 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:00:49.392454 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:00:49.392455 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:00:49.392544 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:49.393065 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:00:49.393073 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:00:49.394398 (+ 1325us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:00:49.395799 (+ 1401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:00:49.395804 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:00:49.395806 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:00:49.396012 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:00:49.396249 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:00:49.396250 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:00:49.396250 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:00:49.396377 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:00:49.396609 (+ 232us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:00:49.396613 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:00:49.396715 (+ 102us) client_negotiation.cc:769] Sending connection context
0929 19:00:49.396794 (+ 79us) client_negotiation.cc:240] Negotiation successful
0929 19:00:49.397004 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":127,"threads_started":1}
I20260929 19:00:49.397614 769 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36981
I20260929 19:00:49.397712 769 heartbeater.cc:464] Registering TS with master...
I20260929 19:00:49.397797 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUbnLHE/build/release/bin/kudu as pid 640
I20260929 19:00:49.397914 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708447806542-392-0/minicluster-data/ts-1/wal/instance
I20260929 19:00:49.397917 769 heartbeater.cc:511] Master 127.0.98.62:36981 requested a full tablet report, sending...
I20260929 19:00:49.398097 392 external_mini_cluster.cc:1428] Setting key 154f8c1bfaa9006ed69c6b159351b7b9
I20260929 19:00:49.398276 444 ts_manager.cc:194] Registered new tserver with Master: e3e00192fbf74d859664323411380e39 (127.0.98.2:41775)
I20260929 19:00:49.398705 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:48379
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708447806542-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708447806542-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260929 19:00:49.419108 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUbnLHE/build/release/bin/kudu
/tmp/dist-test-taskUbnLHE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708447806542-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708447806542-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708447806542-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708447806542-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36981
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708447806542-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708447806542-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708447806542-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708447806542-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:00:49.499150 774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:00:49.499418 774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:00:49.499456 774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:00:49.500599 774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:00:49.500666 774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:00:49.500710 774 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 19:00:49.502148 774 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708447806542-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708447806542-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708447806542-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708447806542-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36981
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708447806542-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 18:09:13 UTC on ccycloud.quasar-hpkalf.root.comops.site
build id 4680913
I20260929 19:00:49.502417 774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:00:49.502635 774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:00:49.507257 780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:00:49.507346 779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:00:49.507346 782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:00:49.507723 774 server_base.cc:1034] running on GCE node
I20260929 19:00:49.507913 774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:00:49.508177 774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:00:49.509339 774 hybrid_clock.cc:648] HybridClock initialized: now 1790708449509330 us; error 33 us; skew 500 ppm
Sep 29 19:00:49 dist-test-slave-lqkv krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708449, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:00:49.511756 774 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 19:00:49.512198 774 webserver.cc:460] Webserver started at http://127.0.98.3:42857/ using document root <none> and password file <none>
I20260929 19:00:49.512442 774 fs_manager.cc:359] Metadata directory not provided
I20260929 19:00:49.512512 774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:00:49.512653 774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:00:49.513715 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708447806542-392-0/minicluster-data/ts-2/data/instance:
uuid: "625667198da844b3b93afb5963a50e55"
format_stamp: "Formatted at 2026-09-29 19:00:49 on dist-test-slave-lqkv"
server_key: "e12185a78be5d7d2f9c858cab68a3f40"
server_key_iv: "ff75f78c2d94b5217eab554b2d75c1aa"
server_key_version: "encryptionkey@0"
I20260929 19:00:49.514076 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708447806542-392-0/minicluster-data/ts-2/wal/instance:
uuid: "625667198da844b3b93afb5963a50e55"
format_stamp: "Formatted at 2026-09-29 19:00:49 on dist-test-slave-lqkv"
server_key: "e12185a78be5d7d2f9c858cab68a3f40"
server_key_iv: "ff75f78c2d94b5217eab554b2d75c1aa"
server_key_version: "encryptionkey@0"
I20260929 19:00:49.515375 774 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 19:00:49.516108 789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:00:49.516305 774 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 19:00:49.516407 774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708447806542-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708447806542-392-0/minicluster-data/ts-2/wal
uuid: "625667198da844b3b93afb5963a50e55"
format_stamp: "Formatted at 2026-09-29 19:00:49 on dist-test-slave-lqkv"
server_key: "e12185a78be5d7d2f9c858cab68a3f40"
server_key_iv: "ff75f78c2d94b5217eab554b2d75c1aa"
server_key_version: "encryptionkey@0"
I20260929 19:00:49.516489 774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708447806542-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708447806542-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708447806542-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:00:49.528829 774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:00:49.531018 774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:00:49.531167 774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:00:49.531769 774 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:00:49.532058 774 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:00:49.532110 774 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:00:49.532150 774 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:00:49.532181 774 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:00:49.538589 774 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39817
I20260929 19:00:49.538631 902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39817 every 8 connection(s)
I20260929 19:00:49.538918 774 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708447806542-392-0/minicluster-data/ts-2/data/info.pb
I20260929 19:00:49.543149 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUbnLHE/build/release/bin/kudu as pid 774
I20260929 19:00:49.543227 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790708447806542-392-0/minicluster-data/ts-2/wal/instance
I20260929 19:00:49.543398 392 external_mini_cluster.cc:1428] Setting key cb0baf8da1cffdf8d3e272e09ca0156a
Sep 29 19:00:49 dist-test-slave-lqkv krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708449, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 19:00:49.547472 483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:49.539492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50381 (local address 127.0.98.62:36981)
0929 19:00:49.539672 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0929 19:00:49.539674 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:00:49.539760 (+ 86us) server_negotiation.cc:407] Connection header received
0929 19:00:49.540282 (+ 522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:00:49.540283 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:00:49.540305 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:00:49.540354 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:00:49.541347 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:49.541840 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:49.542801 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:49.542992 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:49.543062 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:49.545271 (+ 2209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:00:49.545272 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:00:49.545273 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:00:49.545278 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:00:49.546424 (+ 1146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:00:49.546762 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:00:49.546763 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:00:49.546763 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:00:49.546815 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:00:49.547018 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:00:49.547018 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:00:49.547019 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:00:49.547109 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:00:49.547157 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0929 19:00:49.547359 (+ 202us) server_negotiation.cc:299] Negotiation successful
0929 19:00:49.547413 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158}
I20260929 19:00:49.547633 905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:49.539491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36981 (local address 127.0.98.3:50381)
0929 19:00:49.539640 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0929 19:00:49.539674 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0929 19:00:49.540177 (+ 503us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:00:49.540418 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:00:49.540422 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:00:49.541068 (+ 646us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:00:49.541245 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:00:49.541250 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:00:49.541929 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:00:49.541930 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:00:49.542701 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:00:49.542703 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:00:49.543078 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:00:49.543079 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:00:49.543238 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:49.543770 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:00:49.543786 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:00:49.545197 (+ 1411us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:00:49.546491 (+ 1294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:00:49.546496 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:00:49.546497 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:00:49.546700 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:00:49.546872 (+ 172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:00:49.546873 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:00:49.546873 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:00:49.546966 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:00:49.547161 (+ 195us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:00:49.547165 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:00:49.547266 (+ 101us) client_negotiation.cc:769] Sending connection context
0929 19:00:49.547340 (+ 74us) client_negotiation.cc:240] Negotiation successful
0929 19:00:49.547453 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":72,"threads_started":1}
I20260929 19:00:49.548094 903 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36981
I20260929 19:00:49.548163 903 heartbeater.cc:464] Registering TS with master...
I20260929 19:00:49.548343 903 heartbeater.cc:511] Master 127.0.98.62:36981 requested a full tablet report, sending...
I20260929 19:00:49.548698 444 ts_manager.cc:194] Registered new tserver with Master: 625667198da844b3b93afb5963a50e55 (127.0.98.3:39817)
I20260929 19:00:49.549088 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:50381
I20260929 19:00:49.555330 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 19:00:49 dist-test-slave-lqkv krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708449, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 19:00:49 dist-test-slave-lqkv krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708449, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260929 19:00:49.576568 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:49.569379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53890 (local address 127.0.98.1:37773)
0929 19:00:49.569541 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0929 19:00:49.569545 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 19:00:49.569555 (+ 10us) server_negotiation.cc:407] Connection header received
0929 19:00:49.569577 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:00:49.569579 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:00:49.569591 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:00:49.569650 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:00:49.570164 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:49.570841 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:49.571613 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:49.571908 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:49.571961 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:49.574201 (+ 2240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:00:49.574207 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:00:49.574208 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:00:49.574213 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:00:49.575700 (+ 1487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:00:49.575990 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:00:49.575993 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:00:49.575993 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:00:49.576043 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:00:49.576182 (+ 139us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:00:49.576183 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:00:49.576183 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:00:49.576284 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:00:49.576377 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0929 19:00:49.576443 (+ 66us) server_negotiation.cc:299] Negotiation successful
0929 19:00:49.576498 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":84,"threads_started":1}
W20260929 19:00:49.576946 549 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53890
Sep 29 19:00:49 dist-test-slave-lqkv krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708449, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 19:00:49 dist-test-slave-lqkv krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708449, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260929 19:00:49.597760 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:49.590989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53902 (local address 127.0.98.1:37773)
0929 19:00:49.591054 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0929 19:00:49.591056 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:00:49.591133 (+ 77us) server_negotiation.cc:407] Connection header received
0929 19:00:49.591231 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:00:49.591232 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:00:49.591245 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:00:49.591315 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:00:49.591834 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:49.592300 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:49.593123 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:49.593323 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:49.593429 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:49.595458 (+ 2029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:00:49.595459 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:00:49.595459 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:00:49.595466 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:00:49.596691 (+ 1225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:00:49.597050 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:00:49.597051 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:00:49.597051 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:00:49.597103 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:00:49.597291 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:00:49.597292 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:00:49.597292 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:00:49.597389 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:00:49.597504 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0929 19:00:49.597589 (+ 85us) server_negotiation.cc:299] Negotiation successful
0929 19:00:49.597662 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260929 19:00:49.598289 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUbnLHE/build/release/bin/kudu with pid 506
I20260929 19:00:49.602844 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUbnLHE/build/release/bin/kudu with pid 640
I20260929 19:00:49.608182 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUbnLHE/build/release/bin/kudu with pid 774
I20260929 19:00:49.612313 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUbnLHE/build/release/bin/kudu with pid 412
2026-09-29T19:00:49Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (1812 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708447806542-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 19:00:49 dist-test-slave-lqkv krb5kdc[917](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 19:00:49 dist-test-slave-lqkv krb5kdc[917](info): set up 2 sockets
Sep 29 19:00:49 dist-test-slave-lqkv krb5kdc[917](info): commencing operation
krb5kdc: starting...
W20260929 19:00:50.628230 392 mini_kdc.cc:121] Time spent starting KDC: real 0.996s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708447806542-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708447806542-392-0/krb5kdc/test-user.keytab.
Sep 29 19:00:50 dist-test-slave-lqkv krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708450, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T19:00:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T19:00:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708447806542-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708447806542-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 19:00:50.735249 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUbnLHE/build/release/bin/kudu
/tmp/dist-test-taskUbnLHE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708447806542-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708447806542-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708447806542-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708447806542-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33747
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708447806542-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33747
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708447806542-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708447806542-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708447806542-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:00:50.812793 930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:00:50.813032 930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:00:50.813053 930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:00:50.814090 930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 19:00:50.814126 930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:00:50.814139 930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:00:50.814186 930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 19:00:50.814201 930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 19:00:50.815579 930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708447806542-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708447806542-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33747
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708447806542-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33747
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708447806542-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708447806542-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 18:09:13 UTC on ccycloud.quasar-hpkalf.root.comops.site
build id 4680913
I20260929 19:00:50.815809 930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:00:50.816021 930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:00:50.820122 935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:00:50.820154 938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:00:50.820114 936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:00:50.820327 930 server_base.cc:1034] running on GCE node
I20260929 19:00:50.820528 930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:00:50.820830 930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:00:50.821969 930 hybrid_clock.cc:648] HybridClock initialized: now 1790708450821957 us; error 26 us; skew 500 ppm
Sep 29 19:00:50 dist-test-slave-lqkv krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708450, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:00:50.824191 930 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 19:00:50.824546 930 webserver.cc:460] Webserver started at http://127.0.98.62:43947/ using document root <none> and password file <none>
I20260929 19:00:50.824759 930 fs_manager.cc:359] Metadata directory not provided
I20260929 19:00:50.824846 930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:00:50.824996 930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:00:50.825968 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708447806542-392-0/minicluster-data/master-0/data/instance:
uuid: "341cd1d47a544890bec5c17f97a44bcc"
format_stamp: "Formatted at 2026-09-29 19:00:50 on dist-test-slave-lqkv"
server_key: "49fa79dd07f0599bcf668088c0bcd307"
server_key_iv: "9a4d4663c062bed8c1f9cf9aad911a5f"
server_key_version: "encryptionkey@0"
I20260929 19:00:50.826288 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708447806542-392-0/minicluster-data/master-0/wal/instance:
uuid: "341cd1d47a544890bec5c17f97a44bcc"
format_stamp: "Formatted at 2026-09-29 19:00:50 on dist-test-slave-lqkv"
server_key: "49fa79dd07f0599bcf668088c0bcd307"
server_key_iv: "9a4d4663c062bed8c1f9cf9aad911a5f"
server_key_version: "encryptionkey@0"
I20260929 19:00:50.827469 930 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260929 19:00:50.828117 945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:00:50.828315 930 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 19:00:50.828431 930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708447806542-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708447806542-392-0/minicluster-data/master-0/wal
uuid: "341cd1d47a544890bec5c17f97a44bcc"
format_stamp: "Formatted at 2026-09-29 19:00:50 on dist-test-slave-lqkv"
server_key: "49fa79dd07f0599bcf668088c0bcd307"
server_key_iv: "9a4d4663c062bed8c1f9cf9aad911a5f"
server_key_version: "encryptionkey@0"
I20260929 19:00:50.828497 930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708447806542-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708447806542-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708447806542-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:00:50.837564 930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:00:50.839669 930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:00:50.839829 930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:00:50.843429 930 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33747
I20260929 19:00:50.843488 997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33747 every 8 connection(s)
I20260929 19:00:50.843766 930 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708447806542-392-0/minicluster-data/master-0/data/info.pb
I20260929 19:00:50.844561 998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:00:50.846576 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 341cd1d47a544890bec5c17f97a44bcc: Bootstrap starting.
I20260929 19:00:50.847040 998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 341cd1d47a544890bec5c17f97a44bcc: Neither blocks nor log segments found. Creating new log.
I20260929 19:00:50.847306 998 log.cc:824] T 00000000000000000000000000000000 P 341cd1d47a544890bec5c17f97a44bcc: Log is configured to *not* fsync() on all Append() calls
I20260929 19:00:50.847896 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 341cd1d47a544890bec5c17f97a44bcc: No bootstrap required, opened a new log
I20260929 19:00:50.849300 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUbnLHE/build/release/bin/kudu as pid 930
I20260929 19:00:50.849385 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708447806542-392-0/minicluster-data/master-0/wal/instance
I20260929 19:00:50.849534 392 external_mini_cluster.cc:1428] Setting key 63d053f72dda73b1e54caaa2ea96f92d
I20260929 19:00:50.849615 998 raft_consensus.cc:348] T 00000000000000000000000000000000 P 341cd1d47a544890bec5c17f97a44bcc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "341cd1d47a544890bec5c17f97a44bcc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33747 } }
I20260929 19:00:50.849759 998 raft_consensus.cc:374] T 00000000000000000000000000000000 P 341cd1d47a544890bec5c17f97a44bcc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:00:50.849804 998 raft_consensus.cc:729] T 00000000000000000000000000000000 P 341cd1d47a544890bec5c17f97a44bcc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 341cd1d47a544890bec5c17f97a44bcc, State: Initialized, Role: FOLLOWER
I20260929 19:00:50.849937 998 consensus_queue.cc:260] T 00000000000000000000000000000000 P 341cd1d47a544890bec5c17f97a44bcc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "341cd1d47a544890bec5c17f97a44bcc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33747 } }
I20260929 19:00:50.850019 998 raft_consensus.cc:388] T 00000000000000000000000000000000 P 341cd1d47a544890bec5c17f97a44bcc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 19:00:50.850066 998 raft_consensus.cc:482] T 00000000000000000000000000000000 P 341cd1d47a544890bec5c17f97a44bcc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 19:00:50.850107 998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 341cd1d47a544890bec5c17f97a44bcc [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:00:50.851032 998 raft_consensus.cc:504] T 00000000000000000000000000000000 P 341cd1d47a544890bec5c17f97a44bcc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "341cd1d47a544890bec5c17f97a44bcc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33747 } }
I20260929 19:00:50.851126 998 leader_election.cc:302] T 00000000000000000000000000000000 P 341cd1d47a544890bec5c17f97a44bcc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 341cd1d47a544890bec5c17f97a44bcc; no voters:
I20260929 19:00:50.851265 998 leader_election.cc:288] T 00000000000000000000000000000000 P 341cd1d47a544890bec5c17f97a44bcc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 19:00:50.851323 1003 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 341cd1d47a544890bec5c17f97a44bcc [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:00:50.851476 998 sys_catalog.cc:564] T 00000000000000000000000000000000 P 341cd1d47a544890bec5c17f97a44bcc [sys.catalog]: configured and running, proceeding with master startup.
I20260929 19:00:50.851464 1003 raft_consensus.cc:686] T 00000000000000000000000000000000 P 341cd1d47a544890bec5c17f97a44bcc [term 1 LEADER]: Becoming Leader. State: Replica: 341cd1d47a544890bec5c17f97a44bcc, State: Running, Role: LEADER
I20260929 19:00:50.851621 1003 consensus_queue.cc:237] T 00000000000000000000000000000000 P 341cd1d47a544890bec5c17f97a44bcc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "341cd1d47a544890bec5c17f97a44bcc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33747 } }
I20260929 19:00:50.851912 1005 sys_catalog.cc:455] T 00000000000000000000000000000000 P 341cd1d47a544890bec5c17f97a44bcc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "341cd1d47a544890bec5c17f97a44bcc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "341cd1d47a544890bec5c17f97a44bcc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33747 } } }
I20260929 19:00:50.852005 1005 sys_catalog.cc:458] T 00000000000000000000000000000000 P 341cd1d47a544890bec5c17f97a44bcc [sys.catalog]: This master's current role is: LEADER
I20260929 19:00:50.851903 1003 sys_catalog.cc:455] T 00000000000000000000000000000000 P 341cd1d47a544890bec5c17f97a44bcc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 341cd1d47a544890bec5c17f97a44bcc. Latest consensus state: current_term: 1 leader_uuid: "341cd1d47a544890bec5c17f97a44bcc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "341cd1d47a544890bec5c17f97a44bcc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33747 } } }
I20260929 19:00:50.852169 1003 sys_catalog.cc:458] T 00000000000000000000000000000000 P 341cd1d47a544890bec5c17f97a44bcc [sys.catalog]: This master's current role is: LEADER
I20260929 19:00:50.852298 1012 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 19:00:50.852943 1012 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 19:00:50.854488 1012 catalog_manager.cc:1347] Generated new cluster ID: 2c5964dc0a354572b21adeab65af54b2
I20260929 19:00:50.854533 1012 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 29 19:00:50 dist-test-slave-lqkv krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708450, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 19:00:50.860229 1002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:50.849994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44002 (local address 127.0.98.62:33747)
0929 19:00:50.850155 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0929 19:00:50.850163 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 19:00:50.850192 (+ 29us) server_negotiation.cc:407] Connection header received
0929 19:00:50.850314 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:00:50.850321 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:00:50.850381 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:00:50.850542 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:00:50.851056 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:50.852471 (+ 1415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:50.853502 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:50.853957 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:50.854054 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:50.856790 (+ 2736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:00:50.856793 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:00:50.856793 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:00:50.856822 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:00:50.858641 (+ 1819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:00:50.859040 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:00:50.859043 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:00:50.859043 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:00:50.859146 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:00:50.859411 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:00:50.859412 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:00:50.859412 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:00:50.859699 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:00:50.859862 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0929 19:00:50.859926 (+ 64us) server_negotiation.cc:299] Negotiation successful
0929 19:00:50.860051 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":78,"threads_started":1}
I20260929 19:00:50.875102 1012 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 19:00:50.875779 1012 catalog_manager.cc:1504] Loading token signing keys...
I20260929 19:00:50.880774 1012 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 341cd1d47a544890bec5c17f97a44bcc: Generated new TSK 0
I20260929 19:00:50.880986 1012 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708447806542-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708447806542-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260929 19:00:50.905557 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUbnLHE/build/release/bin/kudu
/tmp/dist-test-taskUbnLHE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708447806542-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708447806542-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708447806542-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708447806542-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33747
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708447806542-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708447806542-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708447806542-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708447806542-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:00:50.982290 1024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:00:50.982468 1024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:00:50.982489 1024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:00:50.983502 1024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:00:50.983532 1024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:00:50.983557 1024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 19:00:50.984859 1024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708447806542-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708447806542-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708447806542-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708447806542-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33747
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708447806542-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 18:09:13 UTC on ccycloud.quasar-hpkalf.root.comops.site
build id 4680913
I20260929 19:00:50.985014 1024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:00:50.985170 1024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:00:50.989691 1032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:00:50.989766 1029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:00:50.989898 1024 server_base.cc:1034] running on GCE node
W20260929 19:00:50.989930 1030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:00:50.990119 1024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:00:50.990386 1024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:00:50.991549 1024 hybrid_clock.cc:648] HybridClock initialized: now 1790708450991529 us; error 34 us; skew 500 ppm
Sep 29 19:00:50 dist-test-slave-lqkv krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708450, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:00:50.993920 1024 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 19:00:50.994355 1024 webserver.cc:460] Webserver started at http://127.0.98.1:32951/ using document root <none> and password file <none>
I20260929 19:00:50.994566 1024 fs_manager.cc:359] Metadata directory not provided
I20260929 19:00:50.994632 1024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:00:50.994786 1024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:00:50.995776 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708447806542-392-0/minicluster-data/ts-0/data/instance:
uuid: "01f9ff70a77042fbb07ba630f2604290"
format_stamp: "Formatted at 2026-09-29 19:00:50 on dist-test-slave-lqkv"
server_key: "74c222c4640857bf948b0cdd1a554833"
server_key_iv: "60a6951444fc023d528eaa3ccdc18712"
server_key_version: "encryptionkey@0"
I20260929 19:00:50.996125 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708447806542-392-0/minicluster-data/ts-0/wal/instance:
uuid: "01f9ff70a77042fbb07ba630f2604290"
format_stamp: "Formatted at 2026-09-29 19:00:50 on dist-test-slave-lqkv"
server_key: "74c222c4640857bf948b0cdd1a554833"
server_key_iv: "60a6951444fc023d528eaa3ccdc18712"
server_key_version: "encryptionkey@0"
I20260929 19:00:50.997473 1024 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 19:00:50.998157 1039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:00:50.998319 1024 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 19:00:50.998412 1024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708447806542-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708447806542-392-0/minicluster-data/ts-0/wal
uuid: "01f9ff70a77042fbb07ba630f2604290"
format_stamp: "Formatted at 2026-09-29 19:00:50 on dist-test-slave-lqkv"
server_key: "74c222c4640857bf948b0cdd1a554833"
server_key_iv: "60a6951444fc023d528eaa3ccdc18712"
server_key_version: "encryptionkey@0"
I20260929 19:00:50.998489 1024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708447806542-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708447806542-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708447806542-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:00:51.016711 1024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:00:51.018589 1024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:00:51.018766 1024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:00:51.018973 1024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:00:51.019254 1024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:00:51.019304 1024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:00:51.019351 1024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:00:51.019377 1024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:00:51.025028 1024 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44145
I20260929 19:00:51.025074 1152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44145 every 8 connection(s)
I20260929 19:00:51.025406 1024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708447806542-392-0/minicluster-data/ts-0/data/info.pb
I20260929 19:00:51.029660 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUbnLHE/build/release/bin/kudu as pid 1024
I20260929 19:00:51.029752 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708447806542-392-0/minicluster-data/ts-0/wal/instance
I20260929 19:00:51.029927 392 external_mini_cluster.cc:1428] Setting key 5ee808ee4e227d95bea126f7307f6219
Sep 29 19:00:51 dist-test-slave-lqkv krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708450, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 19:00:51.035022 1002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:51.026094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39689 (local address 127.0.98.62:33747)
0929 19:00:51.026157 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0929 19:00:51.026159 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:00:51.026367 (+ 208us) server_negotiation.cc:407] Connection header received
0929 19:00:51.026814 (+ 447us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:00:51.026814 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:00:51.026822 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:00:51.026875 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:00:51.027727 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:51.028220 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:51.029216 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:51.029405 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:51.029459 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:51.032182 (+ 2723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:00:51.032184 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:00:51.032184 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:00:51.032197 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:00:51.033681 (+ 1484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:00:51.034092 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:00:51.034093 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:00:51.034094 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:00:51.034152 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:00:51.034388 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:00:51.034389 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:00:51.034390 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:00:51.034501 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:00:51.034573 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 19:00:51.034843 (+ 270us) server_negotiation.cc:299] Negotiation successful
0929 19:00:51.034929 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260929 19:00:51.035213 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:51.026084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33747 (local address 127.0.98.1:39689)
0929 19:00:51.026246 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0929 19:00:51.026285 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 19:00:51.026737 (+ 452us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:00:51.026927 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:00:51.026930 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:00:51.027470 (+ 540us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:00:51.027650 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:00:51.027655 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:00:51.028294 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:00:51.028294 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:00:51.029145 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:00:51.029147 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:00:51.029521 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:00:51.029522 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:00:51.029678 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:51.030327 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:00:51.030344 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:00:51.032063 (+ 1719us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:00:51.033788 (+ 1725us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:00:51.033794 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:00:51.033797 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:00:51.034008 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:00:51.034217 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:00:51.034218 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:00:51.034219 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:00:51.034322 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:00:51.034565 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:00:51.034570 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:00:51.034713 (+ 143us) client_negotiation.cc:769] Sending connection context
0929 19:00:51.034820 (+ 107us) client_negotiation.cc:240] Negotiation successful
0929 19:00:51.034973 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":82,"threads_started":1}
I20260929 19:00:51.035878 1153 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33747
I20260929 19:00:51.035956 1153 heartbeater.cc:464] Registering TS with master...
I20260929 19:00:51.036141 1153 heartbeater.cc:511] Master 127.0.98.62:33747 requested a full tablet report, sending...
I20260929 19:00:51.036659 962 ts_manager.cc:194] Registered new tserver with Master: 01f9ff70a77042fbb07ba630f2604290 (127.0.98.1:44145)
I20260929 19:00:51.037283 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:39689
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708447806542-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708447806542-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260929 19:00:51.053516 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUbnLHE/build/release/bin/kudu
/tmp/dist-test-taskUbnLHE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708447806542-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708447806542-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708447806542-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708447806542-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33747
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708447806542-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708447806542-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708447806542-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708447806542-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:00:51.129063 1158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:00:51.129230 1158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:00:51.129251 1158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:00:51.130324 1158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:00:51.130363 1158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:00:51.130411 1158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 19:00:51.131897 1158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708447806542-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708447806542-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708447806542-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708447806542-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33747
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708447806542-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 18:09:13 UTC on ccycloud.quasar-hpkalf.root.comops.site
build id 4680913
I20260929 19:00:51.132128 1158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:00:51.132347 1158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:00:51.137050 1163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:00:51.137104 1158 server_base.cc:1034] running on GCE node
W20260929 19:00:51.137033 1164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:00:51.137035 1166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:00:51.137346 1158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:00:51.137523 1158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:00:51.138698 1158 hybrid_clock.cc:648] HybridClock initialized: now 1790708451138682 us; error 26 us; skew 500 ppm
Sep 29 19:00:51 dist-test-slave-lqkv krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708451, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:00:51.141033 1158 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 19:00:51.141448 1158 webserver.cc:460] Webserver started at http://127.0.98.2:36627/ using document root <none> and password file <none>
I20260929 19:00:51.141662 1158 fs_manager.cc:359] Metadata directory not provided
I20260929 19:00:51.141737 1158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:00:51.141867 1158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:00:51.142803 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708447806542-392-0/minicluster-data/ts-1/data/instance:
uuid: "8463b5f49a1943efb9de0f073def609a"
format_stamp: "Formatted at 2026-09-29 19:00:51 on dist-test-slave-lqkv"
server_key: "e50173a3db68809994c06013bf314db1"
server_key_iv: "5a19a49fed23492564548154c6e0e35a"
server_key_version: "encryptionkey@0"
I20260929 19:00:51.143137 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708447806542-392-0/minicluster-data/ts-1/wal/instance:
uuid: "8463b5f49a1943efb9de0f073def609a"
format_stamp: "Formatted at 2026-09-29 19:00:51 on dist-test-slave-lqkv"
server_key: "e50173a3db68809994c06013bf314db1"
server_key_iv: "5a19a49fed23492564548154c6e0e35a"
server_key_version: "encryptionkey@0"
I20260929 19:00:51.144354 1158 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 19:00:51.145112 1173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:00:51.145291 1158 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 19:00:51.145375 1158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708447806542-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708447806542-392-0/minicluster-data/ts-1/wal
uuid: "8463b5f49a1943efb9de0f073def609a"
format_stamp: "Formatted at 2026-09-29 19:00:51 on dist-test-slave-lqkv"
server_key: "e50173a3db68809994c06013bf314db1"
server_key_iv: "5a19a49fed23492564548154c6e0e35a"
server_key_version: "encryptionkey@0"
I20260929 19:00:51.145452 1158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708447806542-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708447806542-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708447806542-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:00:51.155112 1158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:00:51.157207 1158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:00:51.157342 1158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:00:51.157526 1158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:00:51.157774 1158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:00:51.157822 1158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:00:51.157858 1158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:00:51.157883 1158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:00:51.163180 1158 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33295
I20260929 19:00:51.163228 1286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33295 every 8 connection(s)
I20260929 19:00:51.163471 1158 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708447806542-392-0/minicluster-data/ts-1/data/info.pb
I20260929 19:00:51.167557 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUbnLHE/build/release/bin/kudu as pid 1158
I20260929 19:00:51.167652 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708447806542-392-0/minicluster-data/ts-1/wal/instance
I20260929 19:00:51.167820 392 external_mini_cluster.cc:1428] Setting key cf2b5989f142aab3beea4a39951b679b
Sep 29 19:00:51 dist-test-slave-lqkv krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708451, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 19:00:51.172284 1002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:51.164071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37139 (local address 127.0.98.62:33747)
0929 19:00:51.164115 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0929 19:00:51.164118 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 19:00:51.164312 (+ 194us) server_negotiation.cc:407] Connection header received
0929 19:00:51.164700 (+ 388us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:00:51.164701 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:00:51.164711 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:00:51.164757 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:00:51.165609 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:51.166100 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:51.166960 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:51.167137 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:51.167185 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:51.169218 (+ 2033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:00:51.169220 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:00:51.169220 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:00:51.169227 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:00:51.170840 (+ 1613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:00:51.171200 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:00:51.171201 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:00:51.171202 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:00:51.171266 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:00:51.171537 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:00:51.171538 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:00:51.171538 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:00:51.171639 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:00:51.171695 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0929 19:00:51.172047 (+ 352us) server_negotiation.cc:299] Negotiation successful
0929 19:00:51.172153 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260929 19:00:51.172765 1289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:51.164050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33747 (local address 127.0.98.2:37139)
0929 19:00:51.164202 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0929 19:00:51.164230 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0929 19:00:51.164608 (+ 378us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:00:51.164873 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:00:51.164876 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:00:51.165366 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:00:51.165532 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:00:51.165536 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:00:51.166172 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:00:51.166173 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:00:51.166895 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:00:51.166896 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:00:51.167202 (+ 306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:00:51.167203 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:00:51.167286 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:51.167736 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:00:51.167744 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:00:51.169115 (+ 1371us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:00:51.170915 (+ 1800us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:00:51.170920 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:00:51.170921 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:00:51.171111 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:00:51.171344 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:00:51.171345 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:00:51.171346 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:00:51.171473 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:00:51.171722 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:00:51.171727 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:00:51.171879 (+ 152us) client_negotiation.cc:769] Sending connection context
0929 19:00:51.172370 (+ 491us) client_negotiation.cc:240] Negotiation successful
0929 19:00:51.172526 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":99,"threads_started":1}
I20260929 19:00:51.173270 1287 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33747
I20260929 19:00:51.173365 1287 heartbeater.cc:464] Registering TS with master...
I20260929 19:00:51.173549 1287 heartbeater.cc:511] Master 127.0.98.62:33747 requested a full tablet report, sending...
I20260929 19:00:51.173882 962 ts_manager.cc:194] Registered new tserver with Master: 8463b5f49a1943efb9de0f073def609a (127.0.98.2:33295)
I20260929 19:00:51.174269 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:37139
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708447806542-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708447806542-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260929 19:00:51.190704 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUbnLHE/build/release/bin/kudu
/tmp/dist-test-taskUbnLHE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708447806542-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708447806542-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708447806542-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708447806542-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33747
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708447806542-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708447806542-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708447806542-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708447806542-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:00:51.266585 1292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:00:51.266847 1292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:00:51.266884 1292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:00:51.268009 1292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:00:51.268069 1292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:00:51.268127 1292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 19:00:51.269552 1292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708447806542-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708447806542-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708447806542-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708447806542-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33747
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708447806542-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 18:09:13 UTC on ccycloud.quasar-hpkalf.root.comops.site
build id 4680913
I20260929 19:00:51.269802 1292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:00:51.270006 1292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 19:00:51.274580 1292 server_base.cc:1034] running on GCE node
W20260929 19:00:51.274560 1300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:00:51.274566 1298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:00:51.274569 1297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:00:51.274967 1292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:00:51.275213 1292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:00:51.276362 1292 hybrid_clock.cc:648] HybridClock initialized: now 1790708451276346 us; error 28 us; skew 500 ppm
Sep 29 19:00:51 dist-test-slave-lqkv krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708451, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:00:51.278704 1292 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 19:00:51.279096 1292 webserver.cc:460] Webserver started at http://127.0.98.3:35321/ using document root <none> and password file <none>
I20260929 19:00:51.279316 1292 fs_manager.cc:359] Metadata directory not provided
I20260929 19:00:51.279381 1292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:00:51.279520 1292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:00:51.280467 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708447806542-392-0/minicluster-data/ts-2/data/instance:
uuid: "580308e5fee4408e852bf3bd8b9b03af"
format_stamp: "Formatted at 2026-09-29 19:00:51 on dist-test-slave-lqkv"
server_key: "7cfd03e8c6f0be76ae6ade29ea4d31d1"
server_key_iv: "2e96583ba610415d666bdbf3f29005fd"
server_key_version: "encryptionkey@0"
I20260929 19:00:51.280792 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708447806542-392-0/minicluster-data/ts-2/wal/instance:
uuid: "580308e5fee4408e852bf3bd8b9b03af"
format_stamp: "Formatted at 2026-09-29 19:00:51 on dist-test-slave-lqkv"
server_key: "7cfd03e8c6f0be76ae6ade29ea4d31d1"
server_key_iv: "2e96583ba610415d666bdbf3f29005fd"
server_key_version: "encryptionkey@0"
I20260929 19:00:51.282020 1292 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260929 19:00:51.282691 1307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:00:51.282855 1292 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 19:00:51.282954 1292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708447806542-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708447806542-392-0/minicluster-data/ts-2/wal
uuid: "580308e5fee4408e852bf3bd8b9b03af"
format_stamp: "Formatted at 2026-09-29 19:00:51 on dist-test-slave-lqkv"
server_key: "7cfd03e8c6f0be76ae6ade29ea4d31d1"
server_key_iv: "2e96583ba610415d666bdbf3f29005fd"
server_key_version: "encryptionkey@0"
I20260929 19:00:51.283026 1292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708447806542-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708447806542-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708447806542-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:00:51.293061 1292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:00:51.295148 1292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:00:51.295303 1292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:00:51.295500 1292 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:00:51.295766 1292 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:00:51.295830 1292 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:00:51.295867 1292 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:00:51.295893 1292 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:00:51.301589 1292 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45165
I20260929 19:00:51.301653 1420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45165 every 8 connection(s)
I20260929 19:00:51.301929 1292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708447806542-392-0/minicluster-data/ts-2/data/info.pb
I20260929 19:00:51.304656 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUbnLHE/build/release/bin/kudu as pid 1292
I20260929 19:00:51.304781 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790708447806542-392-0/minicluster-data/ts-2/wal/instance
I20260929 19:00:51.304962 392 external_mini_cluster.cc:1428] Setting key 56d729c2ecda945c8440f403c0671bfb
Sep 29 19:00:51 dist-test-slave-lqkv krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708451, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 19:00:51.310618 1002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:51.302559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37731 (local address 127.0.98.62:33747)
0929 19:00:51.302622 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0929 19:00:51.302624 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:00:51.302804 (+ 180us) server_negotiation.cc:407] Connection header received
0929 19:00:51.303254 (+ 450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:00:51.303256 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:00:51.303268 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:00:51.303315 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:00:51.304105 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:51.304583 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:51.305962 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:51.306161 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:51.306239 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:51.308176 (+ 1937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:00:51.308177 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:00:51.308178 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:00:51.308184 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:00:51.309380 (+ 1196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:00:51.309767 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:00:51.309768 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:00:51.309769 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:00:51.309825 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:00:51.310045 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:00:51.310046 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:00:51.310046 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:00:51.310155 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:00:51.310226 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 19:00:51.310472 (+ 246us) server_negotiation.cc:299] Negotiation successful
0929 19:00:51.310540 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260929 19:00:51.310765 1423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:51.302544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33747 (local address 127.0.98.3:37731)
0929 19:00:51.302696 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0929 19:00:51.302731 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0929 19:00:51.303147 (+ 416us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:00:51.303371 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:00:51.303374 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:00:51.303856 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:00:51.304013 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:00:51.304017 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:00:51.304673 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:00:51.304674 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:00:51.305860 (+ 1186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:00:51.305862 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:00:51.306246 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:00:51.306247 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:00:51.306368 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:51.306851 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:00:51.306858 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:00:51.308036 (+ 1178us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:00:51.309483 (+ 1447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:00:51.309489 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:00:51.309490 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:00:51.309688 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:00:51.309887 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:00:51.309888 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:00:51.309888 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:00:51.309981 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:00:51.310218 (+ 237us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:00:51.310223 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:00:51.310342 (+ 119us) client_negotiation.cc:769] Sending connection context
0929 19:00:51.310450 (+ 108us) client_negotiation.cc:240] Negotiation successful
0929 19:00:51.310576 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":77,"threads_started":1}
I20260929 19:00:51.311162 1421 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33747
I20260929 19:00:51.311249 1421 heartbeater.cc:464] Registering TS with master...
I20260929 19:00:51.311379 1421 heartbeater.cc:511] Master 127.0.98.62:33747 requested a full tablet report, sending...
I20260929 19:00:51.311691 962 ts_manager.cc:194] Registered new tserver with Master: 580308e5fee4408e852bf3bd8b9b03af (127.0.98.3:45165)
I20260929 19:00:51.312022 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:37731
I20260929 19:00:51.316334 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 19:00:51.323068 1002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:51.317773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44018 (local address 127.0.98.62:33747)
0929 19:00:51.317845 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0929 19:00:51.317847 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:00:51.317938 (+ 91us) server_negotiation.cc:407] Connection header received
0929 19:00:51.318003 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:00:51.318004 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:00:51.318012 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:00:51.318056 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:00:51.318548 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:51.318960 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:51.319654 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:51.319847 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:51.319923 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:51.321094 (+ 1171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:00:51.321095 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:00:51.321095 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:00:51.321101 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:00:51.322083 (+ 982us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:00:51.322378 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:00:51.322379 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:00:51.322379 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:00:51.322424 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:00:51.322608 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:00:51.322609 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:00:51.322609 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:00:51.322736 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:00:51.322827 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0929 19:00:51.322922 (+ 95us) server_negotiation.cc:299] Negotiation successful
0929 19:00:51.322986 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 19:00:51.324985 962 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44018:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 19:00:51.326556 962 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 19:00:51.332448 1430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:51.328399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44145 (local address 127.0.0.1:59858)
0929 19:00:51.328547 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0929 19:00:51.328586 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 19:00:51.328665 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:00:51.328943 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:00:51.328945 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:00:51.328947 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:00:51.329034 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:00:51.329036 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:00:51.330097 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:00:51.330098 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:00:51.331302 (+ 1204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:00:51.331304 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:00:51.332168 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:00:51.332175 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:00:51.332248 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:51.332261 (+ 13us) client_negotiation.cc:769] Sending connection context
0929 19:00:51.332323 (+ 62us) client_negotiation.cc:240] Negotiation successful
0929 19:00:51.332368 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":52,"threads_started":1}
I20260929 19:00:51.332592 1431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:51.328442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41900 (local address 127.0.98.3:45165)
0929 19:00:51.328817 (+ 375us) server_negotiation.cc:206] Beginning negotiation
0929 19:00:51.328822 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 19:00:51.328986 (+ 164us) server_negotiation.cc:407] Connection header received
0929 19:00:51.329037 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:00:51.329040 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:00:51.329054 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:00:51.329130 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:00:51.329284 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:51.330311 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:51.331496 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:51.332132 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:51.332185 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:51.332226 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0929 19:00:51.332443 (+ 217us) server_negotiation.cc:299] Negotiation successful
0929 19:00:51.332516 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":66,"threads_started":1}
I20260929 19:00:51.332635 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:51.328498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59858 (local address 127.0.98.1:44145)
0929 19:00:51.328695 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0929 19:00:51.328700 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 19:00:51.328711 (+ 11us) server_negotiation.cc:407] Connection header received
0929 19:00:51.328751 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:00:51.328752 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:00:51.328769 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:00:51.328868 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:00:51.329119 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:51.329975 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:51.331393 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:51.332083 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:51.332159 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:51.332223 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0929 19:00:51.332501 (+ 278us) server_negotiation.cc:299] Negotiation successful
0929 19:00:51.332538 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":80,"threads_started":1}
I20260929 19:00:51.332773 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:51.328708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54896 (local address 127.0.98.2:33295)
0929 19:00:51.328883 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0929 19:00:51.328889 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 19:00:51.329205 (+ 316us) server_negotiation.cc:407] Connection header received
0929 19:00:51.329328 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:00:51.329331 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:00:51.329347 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:00:51.329417 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:00:51.329585 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:51.330594 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:51.331612 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:51.332251 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:51.332324 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:51.332372 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0929 19:00:51.332640 (+ 268us) server_negotiation.cc:299] Negotiation successful
0929 19:00:51.332683 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":100,"threads_started":1}
I20260929 19:00:51.332578 1429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:51.328588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33295 (local address 127.0.0.1:54896)
0929 19:00:51.329138 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0929 19:00:51.329147 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 19:00:51.329202 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:00:51.329481 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:00:51.329482 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:00:51.329483 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:00:51.329516 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:00:51.329518 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:00:51.330672 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:00:51.330673 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:00:51.331539 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:00:51.331541 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:00:51.332327 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:00:51.332328 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:00:51.332372 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:51.332375 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 19:00:51.332430 (+ 55us) client_negotiation.cc:240] Negotiation successful
0929 19:00:51.332459 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":88,"threads_started":1}
I20260929 19:00:51.333067 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:51.328411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45165 (local address 127.0.0.1:41900)
0929 19:00:51.328909 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0929 19:00:51.328918 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 19:00:51.328980 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:00:51.329140 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:00:51.329141 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:00:51.329143 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:00:51.329204 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:00:51.329206 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:00:51.330405 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:00:51.330406 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:00:51.331416 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:00:51.331418 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:00:51.332257 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:00:51.332258 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:00:51.332360 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:51.332365 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 19:00:51.332422 (+ 57us) client_negotiation.cc:240] Negotiation successful
0929 19:00:51.332459 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"mutex_wait_us":9,"thread_start_us":44,"threads_started":1}
I20260929 19:00:51.333701 1355 tablet_service.cc:1467] Processing CreateTablet for tablet 6c937a88cd69469d803ad7e5de8f5c82 (DEFAULT_TABLE table=test-table [id=889745be3c8d444c99a89c771c36918b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:00:51.333714 1221 tablet_service.cc:1467] Processing CreateTablet for tablet 6c937a88cd69469d803ad7e5de8f5c82 (DEFAULT_TABLE table=test-table [id=889745be3c8d444c99a89c771c36918b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:00:51.333738 1087 tablet_service.cc:1467] Processing CreateTablet for tablet 6c937a88cd69469d803ad7e5de8f5c82 (DEFAULT_TABLE table=test-table [id=889745be3c8d444c99a89c771c36918b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:00:51.334020 1221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c937a88cd69469d803ad7e5de8f5c82. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:00:51.334020 1355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c937a88cd69469d803ad7e5de8f5c82. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:00:51.334046 1087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c937a88cd69469d803ad7e5de8f5c82. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:00:51.336328 1435 tablet_bootstrap.cc:492] T 6c937a88cd69469d803ad7e5de8f5c82 P 01f9ff70a77042fbb07ba630f2604290: Bootstrap starting.
I20260929 19:00:51.337193 1436 tablet_bootstrap.cc:492] T 6c937a88cd69469d803ad7e5de8f5c82 P 580308e5fee4408e852bf3bd8b9b03af: Bootstrap starting.
I20260929 19:00:51.337458 1435 tablet_bootstrap.cc:654] T 6c937a88cd69469d803ad7e5de8f5c82 P 01f9ff70a77042fbb07ba630f2604290: Neither blocks nor log segments found. Creating new log.
I20260929 19:00:51.337795 1436 tablet_bootstrap.cc:654] T 6c937a88cd69469d803ad7e5de8f5c82 P 580308e5fee4408e852bf3bd8b9b03af: Neither blocks nor log segments found. Creating new log.
I20260929 19:00:51.337980 1435 log.cc:824] T 6c937a88cd69469d803ad7e5de8f5c82 P 01f9ff70a77042fbb07ba630f2604290: Log is configured to *not* fsync() on all Append() calls
I20260929 19:00:51.338265 1436 log.cc:824] T 6c937a88cd69469d803ad7e5de8f5c82 P 580308e5fee4408e852bf3bd8b9b03af: Log is configured to *not* fsync() on all Append() calls
I20260929 19:00:51.338567 1437 tablet_bootstrap.cc:492] T 6c937a88cd69469d803ad7e5de8f5c82 P 8463b5f49a1943efb9de0f073def609a: Bootstrap starting.
I20260929 19:00:51.338719 1435 tablet_bootstrap.cc:492] T 6c937a88cd69469d803ad7e5de8f5c82 P 01f9ff70a77042fbb07ba630f2604290: No bootstrap required, opened a new log
I20260929 19:00:51.338783 1435 ts_tablet_manager.cc:1397] T 6c937a88cd69469d803ad7e5de8f5c82 P 01f9ff70a77042fbb07ba630f2604290: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260929 19:00:51.339457 1436 tablet_bootstrap.cc:492] T 6c937a88cd69469d803ad7e5de8f5c82 P 580308e5fee4408e852bf3bd8b9b03af: No bootstrap required, opened a new log
I20260929 19:00:51.339522 1436 ts_tablet_manager.cc:1397] T 6c937a88cd69469d803ad7e5de8f5c82 P 580308e5fee4408e852bf3bd8b9b03af: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 19:00:51.340631 1435 raft_consensus.cc:348] T 6c937a88cd69469d803ad7e5de8f5c82 P 01f9ff70a77042fbb07ba630f2604290 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01f9ff70a77042fbb07ba630f2604290" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44145 } } peers { permanent_uuid: "580308e5fee4408e852bf3bd8b9b03af" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45165 } } peers { permanent_uuid: "8463b5f49a1943efb9de0f073def609a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33295 } }
I20260929 19:00:51.340795 1435 raft_consensus.cc:374] T 6c937a88cd69469d803ad7e5de8f5c82 P 01f9ff70a77042fbb07ba630f2604290 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:00:51.340853 1435 raft_consensus.cc:729] T 6c937a88cd69469d803ad7e5de8f5c82 P 01f9ff70a77042fbb07ba630f2604290 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01f9ff70a77042fbb07ba630f2604290, State: Initialized, Role: FOLLOWER
I20260929 19:00:51.341017 1437 tablet_bootstrap.cc:654] T 6c937a88cd69469d803ad7e5de8f5c82 P 8463b5f49a1943efb9de0f073def609a: Neither blocks nor log segments found. Creating new log.
I20260929 19:00:51.340992 1435 consensus_queue.cc:260] T 6c937a88cd69469d803ad7e5de8f5c82 P 01f9ff70a77042fbb07ba630f2604290 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01f9ff70a77042fbb07ba630f2604290" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44145 } } peers { permanent_uuid: "580308e5fee4408e852bf3bd8b9b03af" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45165 } } peers { permanent_uuid: "8463b5f49a1943efb9de0f073def609a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33295 } }
I20260929 19:00:51.341244 1435 ts_tablet_manager.cc:1428] T 6c937a88cd69469d803ad7e5de8f5c82 P 01f9ff70a77042fbb07ba630f2604290: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 19:00:51.341360 1153 heartbeater.cc:503] Master 127.0.98.62:33747 was elected leader, sending a full tablet report...
I20260929 19:00:51.341284 1436 raft_consensus.cc:348] T 6c937a88cd69469d803ad7e5de8f5c82 P 580308e5fee4408e852bf3bd8b9b03af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01f9ff70a77042fbb07ba630f2604290" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44145 } } peers { permanent_uuid: "580308e5fee4408e852bf3bd8b9b03af" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45165 } } peers { permanent_uuid: "8463b5f49a1943efb9de0f073def609a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33295 } }
I20260929 19:00:51.341427 1436 raft_consensus.cc:374] T 6c937a88cd69469d803ad7e5de8f5c82 P 580308e5fee4408e852bf3bd8b9b03af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:00:51.341467 1436 raft_consensus.cc:729] T 6c937a88cd69469d803ad7e5de8f5c82 P 580308e5fee4408e852bf3bd8b9b03af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 580308e5fee4408e852bf3bd8b9b03af, State: Initialized, Role: FOLLOWER
I20260929 19:00:51.341554 1437 log.cc:824] T 6c937a88cd69469d803ad7e5de8f5c82 P 8463b5f49a1943efb9de0f073def609a: Log is configured to *not* fsync() on all Append() calls
I20260929 19:00:51.341600 1436 consensus_queue.cc:260] T 6c937a88cd69469d803ad7e5de8f5c82 P 580308e5fee4408e852bf3bd8b9b03af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01f9ff70a77042fbb07ba630f2604290" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44145 } } peers { permanent_uuid: "580308e5fee4408e852bf3bd8b9b03af" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45165 } } peers { permanent_uuid: "8463b5f49a1943efb9de0f073def609a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33295 } }
I20260929 19:00:51.341861 1436 ts_tablet_manager.cc:1428] T 6c937a88cd69469d803ad7e5de8f5c82 P 580308e5fee4408e852bf3bd8b9b03af: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260929 19:00:51.341881 1421 heartbeater.cc:503] Master 127.0.98.62:33747 was elected leader, sending a full tablet report...
I20260929 19:00:51.342339 1437 tablet_bootstrap.cc:492] T 6c937a88cd69469d803ad7e5de8f5c82 P 8463b5f49a1943efb9de0f073def609a: No bootstrap required, opened a new log
I20260929 19:00:51.342402 1437 ts_tablet_manager.cc:1397] T 6c937a88cd69469d803ad7e5de8f5c82 P 8463b5f49a1943efb9de0f073def609a: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260929 19:00:51.344079 1437 raft_consensus.cc:348] T 6c937a88cd69469d803ad7e5de8f5c82 P 8463b5f49a1943efb9de0f073def609a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01f9ff70a77042fbb07ba630f2604290" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44145 } } peers { permanent_uuid: "580308e5fee4408e852bf3bd8b9b03af" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45165 } } peers { permanent_uuid: "8463b5f49a1943efb9de0f073def609a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33295 } }
I20260929 19:00:51.344200 1437 raft_consensus.cc:374] T 6c937a88cd69469d803ad7e5de8f5c82 P 8463b5f49a1943efb9de0f073def609a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:00:51.344221 1437 raft_consensus.cc:729] T 6c937a88cd69469d803ad7e5de8f5c82 P 8463b5f49a1943efb9de0f073def609a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8463b5f49a1943efb9de0f073def609a, State: Initialized, Role: FOLLOWER
I20260929 19:00:51.344341 1437 consensus_queue.cc:260] T 6c937a88cd69469d803ad7e5de8f5c82 P 8463b5f49a1943efb9de0f073def609a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01f9ff70a77042fbb07ba630f2604290" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44145 } } peers { permanent_uuid: "580308e5fee4408e852bf3bd8b9b03af" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45165 } } peers { permanent_uuid: "8463b5f49a1943efb9de0f073def609a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33295 } }
I20260929 19:00:51.344594 1437 ts_tablet_manager.cc:1428] T 6c937a88cd69469d803ad7e5de8f5c82 P 8463b5f49a1943efb9de0f073def609a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 19:00:51.344642 1287 heartbeater.cc:503] Master 127.0.98.62:33747 was elected leader, sending a full tablet report...
W20260929 19:00:51.413933 1288 tablet.cc:2369] T 6c937a88cd69469d803ad7e5de8f5c82 P 8463b5f49a1943efb9de0f073def609a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 19:00:51.451001 1442 raft_consensus.cc:482] T 6c937a88cd69469d803ad7e5de8f5c82 P 580308e5fee4408e852bf3bd8b9b03af [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 19:00:51.451170 1442 raft_consensus.cc:504] T 6c937a88cd69469d803ad7e5de8f5c82 P 580308e5fee4408e852bf3bd8b9b03af [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01f9ff70a77042fbb07ba630f2604290" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44145 } } peers { permanent_uuid: "580308e5fee4408e852bf3bd8b9b03af" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45165 } } peers { permanent_uuid: "8463b5f49a1943efb9de0f073def609a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33295 } }
I20260929 19:00:51.451548 1442 leader_election.cc:288] T 6c937a88cd69469d803ad7e5de8f5c82 P 580308e5fee4408e852bf3bd8b9b03af [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 01f9ff70a77042fbb07ba630f2604290 (127.0.98.1:44145), 8463b5f49a1943efb9de0f073def609a (127.0.98.2:33295)
I20260929 19:00:51.455418 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:51.451791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51013 (local address 127.0.98.1:44145)
0929 19:00:51.451835 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0929 19:00:51.451837 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:00:51.451847 (+ 10us) server_negotiation.cc:407] Connection header received
0929 19:00:51.451930 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:00:51.451931 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:00:51.451944 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:00:51.452000 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:00:51.452211 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:51.452999 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:51.454505 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:51.455081 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:51.455153 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:51.455171 (+ 18us) server_negotiation.cc:1035] Waiting for connection context
0929 19:00:51.455314 (+ 143us) server_negotiation.cc:299] Negotiation successful
0929 19:00:51.455352 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260929 19:00:51.455418 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:51.451948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41223 (local address 127.0.98.2:33295)
0929 19:00:51.451995 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0929 19:00:51.451997 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:00:51.452027 (+ 30us) server_negotiation.cc:407] Connection header received
0929 19:00:51.452082 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:00:51.452083 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:00:51.452096 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:00:51.452163 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:00:51.452323 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:51.453037 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:51.454504 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:51.455088 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:51.455153 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:51.455164 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0929 19:00:51.455314 (+ 150us) server_negotiation.cc:299] Negotiation successful
0929 19:00:51.455352 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260929 19:00:51.455439 1423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:51.451702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44145 (local address 127.0.98.3:51013)
0929 19:00:51.451756 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0929 19:00:51.451784 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0929 19:00:51.451849 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:00:51.452053 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:00:51.452054 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:00:51.452056 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:00:51.452124 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:00:51.452126 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:00:51.453096 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:00:51.453097 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:00:51.454418 (+ 1321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:00:51.454421 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:00:51.455166 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:00:51.455166 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:00:51.455241 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:51.455243 (+ 2us) client_negotiation.cc:769] Sending connection context
0929 19:00:51.455298 (+ 55us) client_negotiation.cc:240] Negotiation successful
0929 19:00:51.455347 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30,"mutex_wait_us":52}
I20260929 19:00:51.455448 1444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:51.451802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33295 (local address 127.0.98.3:41223)
0929 19:00:51.451977 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0929 19:00:51.451986 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 19:00:51.452031 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:00:51.452188 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:00:51.452189 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:00:51.452190 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:00:51.452258 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:00:51.452259 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:00:51.453104 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:00:51.453104 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:00:51.454419 (+ 1315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:00:51.454421 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:00:51.455166 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:00:51.455166 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:00:51.455238 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:51.455242 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 19:00:51.455298 (+ 56us) client_negotiation.cc:240] Negotiation successful
0929 19:00:51.455347 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"mutex_wait_us":83,"thread_start_us":127,"threads_started":1}
I20260929 19:00:51.455821 1241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c937a88cd69469d803ad7e5de8f5c82" candidate_uuid: "580308e5fee4408e852bf3bd8b9b03af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8463b5f49a1943efb9de0f073def609a" is_pre_election: true
I20260929 19:00:51.455822 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c937a88cd69469d803ad7e5de8f5c82" candidate_uuid: "580308e5fee4408e852bf3bd8b9b03af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01f9ff70a77042fbb07ba630f2604290" is_pre_election: true
I20260929 19:00:51.455946 1241 raft_consensus.cc:2454] T 6c937a88cd69469d803ad7e5de8f5c82 P 8463b5f49a1943efb9de0f073def609a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 580308e5fee4408e852bf3bd8b9b03af in term 0.
I20260929 19:00:51.455986 1107 raft_consensus.cc:2454] T 6c937a88cd69469d803ad7e5de8f5c82 P 01f9ff70a77042fbb07ba630f2604290 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 580308e5fee4408e852bf3bd8b9b03af in term 0.
I20260929 19:00:51.456171 1311 leader_election.cc:302] T 6c937a88cd69469d803ad7e5de8f5c82 P 580308e5fee4408e852bf3bd8b9b03af [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 580308e5fee4408e852bf3bd8b9b03af, 8463b5f49a1943efb9de0f073def609a; no voters:
I20260929 19:00:51.456296 1442 raft_consensus.cc:2781] T 6c937a88cd69469d803ad7e5de8f5c82 P 580308e5fee4408e852bf3bd8b9b03af [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 19:00:51.456380 1442 raft_consensus.cc:482] T 6c937a88cd69469d803ad7e5de8f5c82 P 580308e5fee4408e852bf3bd8b9b03af [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 19:00:51.456413 1442 raft_consensus.cc:3037] T 6c937a88cd69469d803ad7e5de8f5c82 P 580308e5fee4408e852bf3bd8b9b03af [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:00:51.457151 1442 raft_consensus.cc:504] T 6c937a88cd69469d803ad7e5de8f5c82 P 580308e5fee4408e852bf3bd8b9b03af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01f9ff70a77042fbb07ba630f2604290" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44145 } } peers { permanent_uuid: "580308e5fee4408e852bf3bd8b9b03af" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45165 } } peers { permanent_uuid: "8463b5f49a1943efb9de0f073def609a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33295 } }
I20260929 19:00:51.457304 1442 leader_election.cc:288] T 6c937a88cd69469d803ad7e5de8f5c82 P 580308e5fee4408e852bf3bd8b9b03af [CANDIDATE]: Term 1 election: Requested vote from peers 01f9ff70a77042fbb07ba630f2604290 (127.0.98.1:44145), 8463b5f49a1943efb9de0f073def609a (127.0.98.2:33295)
I20260929 19:00:51.457445 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c937a88cd69469d803ad7e5de8f5c82" candidate_uuid: "580308e5fee4408e852bf3bd8b9b03af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01f9ff70a77042fbb07ba630f2604290"
I20260929 19:00:51.457472 1241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c937a88cd69469d803ad7e5de8f5c82" candidate_uuid: "580308e5fee4408e852bf3bd8b9b03af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8463b5f49a1943efb9de0f073def609a"
I20260929 19:00:51.457516 1107 raft_consensus.cc:3037] T 6c937a88cd69469d803ad7e5de8f5c82 P 01f9ff70a77042fbb07ba630f2604290 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:00:51.457547 1241 raft_consensus.cc:3037] T 6c937a88cd69469d803ad7e5de8f5c82 P 8463b5f49a1943efb9de0f073def609a [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:00:51.458185 1107 raft_consensus.cc:2454] T 6c937a88cd69469d803ad7e5de8f5c82 P 01f9ff70a77042fbb07ba630f2604290 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 580308e5fee4408e852bf3bd8b9b03af in term 1.
I20260929 19:00:51.458333 1241 raft_consensus.cc:2454] T 6c937a88cd69469d803ad7e5de8f5c82 P 8463b5f49a1943efb9de0f073def609a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 580308e5fee4408e852bf3bd8b9b03af in term 1.
I20260929 19:00:51.458408 1311 leader_election.cc:302] T 6c937a88cd69469d803ad7e5de8f5c82 P 580308e5fee4408e852bf3bd8b9b03af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01f9ff70a77042fbb07ba630f2604290, 580308e5fee4408e852bf3bd8b9b03af; no voters:
I20260929 19:00:51.458482 1442 raft_consensus.cc:2781] T 6c937a88cd69469d803ad7e5de8f5c82 P 580308e5fee4408e852bf3bd8b9b03af [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:00:51.458689 1442 raft_consensus.cc:686] T 6c937a88cd69469d803ad7e5de8f5c82 P 580308e5fee4408e852bf3bd8b9b03af [term 1 LEADER]: Becoming Leader. State: Replica: 580308e5fee4408e852bf3bd8b9b03af, State: Running, Role: LEADER
I20260929 19:00:51.458798 1442 consensus_queue.cc:237] T 6c937a88cd69469d803ad7e5de8f5c82 P 580308e5fee4408e852bf3bd8b9b03af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01f9ff70a77042fbb07ba630f2604290" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44145 } } peers { permanent_uuid: "580308e5fee4408e852bf3bd8b9b03af" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45165 } } peers { permanent_uuid: "8463b5f49a1943efb9de0f073def609a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33295 } }
I20260929 19:00:51.459589 962 catalog_manager.cc:5495] T 6c937a88cd69469d803ad7e5de8f5c82 P 580308e5fee4408e852bf3bd8b9b03af reported cstate change: term changed from 0 to 1, leader changed from <none> to 580308e5fee4408e852bf3bd8b9b03af (127.0.98.3). New cstate: current_term: 1 leader_uuid: "580308e5fee4408e852bf3bd8b9b03af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "01f9ff70a77042fbb07ba630f2604290" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44145 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "580308e5fee4408e852bf3bd8b9b03af" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45165 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8463b5f49a1943efb9de0f073def609a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33295 } health_report { overall_health: UNKNOWN } } }
Sep 29 19:00:51 dist-test-slave-lqkv krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708450, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260929 19:00:51.481040 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:51.474322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59860 (local address 127.0.98.1:44145)
0929 19:00:51.474400 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0929 19:00:51.474402 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:00:51.474454 (+ 52us) server_negotiation.cc:407] Connection header received
0929 19:00:51.474490 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:00:51.474491 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:00:51.474499 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:00:51.474543 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:00:51.475008 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:51.475435 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:51.476060 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:51.476231 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:51.476279 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:51.478495 (+ 2216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:00:51.478503 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:00:51.478503 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:00:51.478512 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:00:51.479863 (+ 1351us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:00:51.480220 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:00:51.480225 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:00:51.480225 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:00:51.480278 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:00:51.480469 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:00:51.480470 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:00:51.480470 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:00:51.480608 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:00:51.480763 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0929 19:00:51.480829 (+ 66us) server_negotiation.cc:299] Negotiation successful
0929 19:00:51.480918 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
Sep 29 19:00:51 dist-test-slave-lqkv krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708451, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 19:00:51 dist-test-slave-lqkv krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708451, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260929 19:00:51.503139 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:51.496257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59864 (local address 127.0.98.1:44145)
0929 19:00:51.496320 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0929 19:00:51.496322 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:00:51.496397 (+ 75us) server_negotiation.cc:407] Connection header received
0929 19:00:51.496479 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:00:51.496484 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:00:51.496493 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:00:51.496562 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:00:51.497129 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:51.497589 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:51.498255 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:51.498445 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:51.498527 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:51.500710 (+ 2183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:00:51.500711 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:00:51.500712 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:00:51.500719 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:00:51.502040 (+ 1321us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:00:51.502390 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:00:51.502391 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:00:51.502391 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:00:51.502440 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:00:51.502626 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:00:51.502626 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:00:51.502627 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:00:51.502721 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:00:51.502813 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0929 19:00:51.502979 (+ 166us) server_negotiation.cc:299] Negotiation successful
0929 19:00:51.503044 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
Sep 29 19:00:51 dist-test-slave-lqkv krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708451, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 19:00:51 dist-test-slave-lqkv krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708451, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260929 19:00:51.523164 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:51.516843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59872 (local address 127.0.98.1:44145)
0929 19:00:51.516887 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0929 19:00:51.516890 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 19:00:51.516987 (+ 97us) server_negotiation.cc:407] Connection header received
0929 19:00:51.517060 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:00:51.517061 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:00:51.517070 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:00:51.517137 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:00:51.517638 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:51.518213 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:51.518877 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:51.519075 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:51.519164 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:51.521223 (+ 2059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:00:51.521224 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:00:51.521224 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:00:51.521230 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:00:51.522327 (+ 1097us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:00:51.522612 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:00:51.522612 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:00:51.522613 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:00:51.522655 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:00:51.522819 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:00:51.522819 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:00:51.522819 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:00:51.522896 (+ 77us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:00:51.522961 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0929 19:00:51.523056 (+ 95us) server_negotiation.cc:299] Negotiation successful
0929 19:00:51.523105 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
W20260929 19:00:51.523597 1067 tablet_service.cc:2908] Rejecting scan request for tablet 6c937a88cd69469d803ad7e5de8f5c82: Uninitialized: clean time has not yet been initialized
I20260929 19:00:51.524379 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUbnLHE/build/release/bin/kudu with pid 1024
I20260929 19:00:51.529073 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUbnLHE/build/release/bin/kudu with pid 1158
I20260929 19:00:51.533492 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUbnLHE/build/release/bin/kudu with pid 1292
I20260929 19:00:51.537411 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUbnLHE/build/release/bin/kudu with pid 930
2026-09-29T19:00:51Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (1926 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708447806542-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 19:00:51 dist-test-slave-lqkv krb5kdc[1459](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 19:00:51 dist-test-slave-lqkv krb5kdc[1459](info): set up 2 sockets
Sep 29 19:00:51 dist-test-slave-lqkv krb5kdc[1459](info): commencing operation
krb5kdc: starting...
W20260929 19:00:52.568375 392 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708447806542-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708447806542-392-0/krb5kdc/test-user.keytab.
Sep 29 19:00:52 dist-test-slave-lqkv krb5kdc[1459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708452, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T19:00:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T19:00:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708447806542-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708447806542-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 19:00:52.679478 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUbnLHE/build/release/bin/kudu
/tmp/dist-test-taskUbnLHE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708447806542-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708447806542-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708447806542-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708447806542-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39367
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708447806542-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39367
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708447806542-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708447806542-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708447806542-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:00:52.755048 1472 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:00:52.755234 1472 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:00:52.755254 1472 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:00:52.756379 1472 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 19:00:52.756418 1472 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260929 19:00:52.756431 1472 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:00:52.756444 1472 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:00:52.756456 1472 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 19:00:52.756469 1472 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 19:00:52.758203 1472 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708447806542-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708447806542-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39367
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708447806542-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39367
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708447806542-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1472
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708447806542-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 18:09:13 UTC on ccycloud.quasar-hpkalf.root.comops.site
build id 4680913
I20260929 19:00:52.758459 1472 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:00:52.758703 1472 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:00:52.763577 1477 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:00:52.763628 1478 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:00:52.763768 1472 server_base.cc:1034] running on GCE node
W20260929 19:00:52.763866 1480 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:00:52.764115 1472 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:00:52.764405 1472 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:00:52.765622 1472 hybrid_clock.cc:648] HybridClock initialized: now 1790708452765598 us; error 57 us; skew 500 ppm
Sep 29 19:00:52 dist-test-slave-lqkv krb5kdc[1459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708452, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:00:52.768579 1472 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 19:00:52.769057 1472 webserver.cc:460] Webserver started at http://127.0.98.62:36219/ using document root <none> and password file <none>
I20260929 19:00:52.769282 1472 fs_manager.cc:359] Metadata directory not provided
I20260929 19:00:52.769347 1472 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:00:52.769476 1472 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:00:52.770505 1472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708447806542-392-0/minicluster-data/master-0/data/instance:
uuid: "4005b06255574d27844b7b7b093be143"
format_stamp: "Formatted at 2026-09-29 19:00:52 on dist-test-slave-lqkv"
server_key: "2baf8625d5201f368d202148eafd693d"
server_key_iv: "a70cf6277960dabd15b54ca0045d541a"
server_key_version: "encryptionkey@0"
I20260929 19:00:52.770853 1472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708447806542-392-0/minicluster-data/master-0/wal/instance:
uuid: "4005b06255574d27844b7b7b093be143"
format_stamp: "Formatted at 2026-09-29 19:00:52 on dist-test-slave-lqkv"
server_key: "2baf8625d5201f368d202148eafd693d"
server_key_iv: "a70cf6277960dabd15b54ca0045d541a"
server_key_version: "encryptionkey@0"
I20260929 19:00:52.772094 1472 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 19:00:52.772756 1487 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:00:52.772953 1472 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 19:00:52.773051 1472 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708447806542-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708447806542-392-0/minicluster-data/master-0/wal
uuid: "4005b06255574d27844b7b7b093be143"
format_stamp: "Formatted at 2026-09-29 19:00:52 on dist-test-slave-lqkv"
server_key: "2baf8625d5201f368d202148eafd693d"
server_key_iv: "a70cf6277960dabd15b54ca0045d541a"
server_key_version: "encryptionkey@0"
I20260929 19:00:52.773101 1472 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708447806542-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708447806542-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708447806542-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:00:52.788177 1472 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:00:52.790289 1472 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:00:52.790387 1472 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:00:52.794543 1472 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39367
I20260929 19:00:52.794615 1549 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39367 every 8 connection(s)
I20260929 19:00:52.794909 1472 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708447806542-392-0/minicluster-data/master-0/data/info.pb
I20260929 19:00:52.795562 1550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:00:52.797711 1550 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4005b06255574d27844b7b7b093be143: Bootstrap starting.
I20260929 19:00:52.798295 1550 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4005b06255574d27844b7b7b093be143: Neither blocks nor log segments found. Creating new log.
I20260929 19:00:52.798593 1550 log.cc:824] T 00000000000000000000000000000000 P 4005b06255574d27844b7b7b093be143: Log is configured to *not* fsync() on all Append() calls
I20260929 19:00:52.799201 1550 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4005b06255574d27844b7b7b093be143: No bootstrap required, opened a new log
I20260929 19:00:52.801232 1550 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4005b06255574d27844b7b7b093be143 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4005b06255574d27844b7b7b093be143" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39367 } }
I20260929 19:00:52.801362 1550 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4005b06255574d27844b7b7b093be143 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:00:52.801384 1550 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4005b06255574d27844b7b7b093be143 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4005b06255574d27844b7b7b093be143, State: Initialized, Role: FOLLOWER
I20260929 19:00:52.801578 1550 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4005b06255574d27844b7b7b093be143 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4005b06255574d27844b7b7b093be143" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39367 } }
I20260929 19:00:52.801654 1550 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4005b06255574d27844b7b7b093be143 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 19:00:52.801697 1550 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4005b06255574d27844b7b7b093be143 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 19:00:52.801753 1550 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4005b06255574d27844b7b7b093be143 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:00:52.802240 1550 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4005b06255574d27844b7b7b093be143 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4005b06255574d27844b7b7b093be143" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39367 } }
I20260929 19:00:52.802356 1550 leader_election.cc:302] T 00000000000000000000000000000000 P 4005b06255574d27844b7b7b093be143 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4005b06255574d27844b7b7b093be143; no voters:
I20260929 19:00:52.802515 1550 leader_election.cc:288] T 00000000000000000000000000000000 P 4005b06255574d27844b7b7b093be143 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 19:00:52.802552 1553 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4005b06255574d27844b7b7b093be143 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:00:52.802690 1553 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4005b06255574d27844b7b7b093be143 [term 1 LEADER]: Becoming Leader. State: Replica: 4005b06255574d27844b7b7b093be143, State: Running, Role: LEADER
I20260929 19:00:52.802778 1550 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4005b06255574d27844b7b7b093be143 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 19:00:52.802815 1553 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4005b06255574d27844b7b7b093be143 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4005b06255574d27844b7b7b093be143" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39367 } }
I20260929 19:00:52.803169 1555 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4005b06255574d27844b7b7b093be143 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4005b06255574d27844b7b7b093be143. Latest consensus state: current_term: 1 leader_uuid: "4005b06255574d27844b7b7b093be143" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4005b06255574d27844b7b7b093be143" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39367 } } }
I20260929 19:00:52.803447 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUbnLHE/build/release/bin/kudu as pid 1472
I20260929 19:00:52.803536 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708447806542-392-0/minicluster-data/master-0/wal/instance
I20260929 19:00:52.803707 392 external_mini_cluster.cc:1428] Setting key 0185ac0fff0a351ca70a0b62c0d74317
I20260929 19:00:52.803161 1554 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4005b06255574d27844b7b7b093be143 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4005b06255574d27844b7b7b093be143" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4005b06255574d27844b7b7b093be143" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39367 } } }
I20260929 19:00:52.804010 1554 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4005b06255574d27844b7b7b093be143 [sys.catalog]: This master's current role is: LEADER
I20260929 19:00:52.804220 1555 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4005b06255574d27844b7b7b093be143 [sys.catalog]: This master's current role is: LEADER
W20260929 19:00:52.804491 1568 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4005b06255574d27844b7b7b093be143: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 19:00:52.804550 1568 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 19:00:52.804603 1569 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 19:00:52.804756 1569 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 19:00:52.806237 1569 catalog_manager.cc:1347] Generated new cluster ID: 81a39467b0dc45658f5a62d479507e2b
I20260929 19:00:52.806288 1569 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 29 19:00:52 dist-test-slave-lqkv krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708452, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 19:00:52.811153 1569 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 19:00:52.811714 1569 catalog_manager.cc:1504] Loading token signing keys...
I20260929 19:00:52.814149 1572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:52.804281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44544 (local address 127.0.98.62:39367)
0929 19:00:52.804692 (+ 411us) server_negotiation.cc:206] Beginning negotiation
0929 19:00:52.804700 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 19:00:52.804729 (+ 29us) server_negotiation.cc:407] Connection header received
0929 19:00:52.804869 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:00:52.804876 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:00:52.804933 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:00:52.805063 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:00:52.805734 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:52.806791 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:52.807541 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:52.807988 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:52.808104 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:52.810372 (+ 2268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:00:52.810376 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:00:52.810376 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:00:52.810386 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:00:52.812359 (+ 1973us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:00:52.812757 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:00:52.812760 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:00:52.812760 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:00:52.812886 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:00:52.813106 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:00:52.813107 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:00:52.813107 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:00:52.813369 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:00:52.813514 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0929 19:00:52.813842 (+ 328us) server_negotiation.cc:299] Negotiation successful
0929 19:00:52.814002 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":57,"threads_started":1}
I20260929 19:00:52.814350 1569 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4005b06255574d27844b7b7b093be143: Generated new TSK 0
I20260929 19:00:52.814481 1569 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708447806542-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708447806542-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260929 19:00:52.843333 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUbnLHE/build/release/bin/kudu
/tmp/dist-test-taskUbnLHE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708447806542-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708447806542-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708447806542-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708447806542-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39367
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708447806542-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708447806542-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708447806542-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708447806542-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:00:52.921164 1576 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:00:52.921342 1576 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:00:52.921365 1576 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:00:52.922500 1576 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:00:52.922545 1576 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:00:52.922565 1576 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 19:00:52.922583 1576 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 19:00:52.924005 1576 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708447806542-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708447806542-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708447806542-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708447806542-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:39367
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1576
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708447806542-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 18:09:13 UTC on ccycloud.quasar-hpkalf.root.comops.site
build id 4680913
I20260929 19:00:52.924211 1576 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:00:52.924412 1576 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:00:52.930104 1584 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:00:52.930121 1582 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:00:52.930331 1581 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:00:52.930310 1576 server_base.cc:1034] running on GCE node
I20260929 19:00:52.930540 1576 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:00:52.930771 1576 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:00:52.931929 1576 hybrid_clock.cc:648] HybridClock initialized: now 1790708452931918 us; error 30 us; skew 500 ppm
Sep 29 19:00:52 dist-test-slave-lqkv krb5kdc[1459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708452, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:00:52.934309 1576 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 19:00:52.934792 1576 webserver.cc:460] Webserver started at http://127.0.98.1:35251/ using document root <none> and password file <none>
I20260929 19:00:52.935012 1576 fs_manager.cc:359] Metadata directory not provided
I20260929 19:00:52.935056 1576 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:00:52.935179 1576 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:00:52.936167 1576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708447806542-392-0/minicluster-data/ts-0/data/instance:
uuid: "ddad36ae0ddf437f9f8c6f86a538b26a"
format_stamp: "Formatted at 2026-09-29 19:00:52 on dist-test-slave-lqkv"
server_key: "16b2f271321e70f7b627e7416ae3ad37"
server_key_iv: "26480c386a8c37653b16c621b78732a3"
server_key_version: "encryptionkey@0"
I20260929 19:00:52.936517 1576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708447806542-392-0/minicluster-data/ts-0/wal/instance:
uuid: "ddad36ae0ddf437f9f8c6f86a538b26a"
format_stamp: "Formatted at 2026-09-29 19:00:52 on dist-test-slave-lqkv"
server_key: "16b2f271321e70f7b627e7416ae3ad37"
server_key_iv: "26480c386a8c37653b16c621b78732a3"
server_key_version: "encryptionkey@0"
I20260929 19:00:52.937863 1576 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 19:00:52.938567 1591 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:00:52.938761 1576 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 19:00:52.938828 1576 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708447806542-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708447806542-392-0/minicluster-data/ts-0/wal
uuid: "ddad36ae0ddf437f9f8c6f86a538b26a"
format_stamp: "Formatted at 2026-09-29 19:00:52 on dist-test-slave-lqkv"
server_key: "16b2f271321e70f7b627e7416ae3ad37"
server_key_iv: "26480c386a8c37653b16c621b78732a3"
server_key_version: "encryptionkey@0"
I20260929 19:00:52.938910 1576 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708447806542-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708447806542-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708447806542-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:00:52.965087 1576 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:00:52.967396 1576 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:00:52.967551 1576 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:00:52.968086 1576 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:00:52.968125 1576 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:00:52.968146 1576 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:00:52.968158 1576 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 19:00:52 dist-test-slave-lqkv krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708452, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 19:00:52.975203 1576 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34477
I20260929 19:00:52.975627 1576 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708447806542-392-0/minicluster-data/ts-0/data/info.pb
I20260929 19:00:52.976048 1706 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34477 every 8 connection(s)
I20260929 19:00:52.977437 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUbnLHE/build/release/bin/kudu as pid 1576
I20260929 19:00:52.977540 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708447806542-392-0/minicluster-data/ts-0/wal/instance
I20260929 19:00:52.977703 392 external_mini_cluster.cc:1428] Setting key 3c98d85b18345add9c0dcd6b40c9871d
I20260929 19:00:52.978024 1572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:52.968477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50803 (local address 127.0.98.62:39367)
0929 19:00:52.968548 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0929 19:00:52.968550 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:00:52.968770 (+ 220us) server_negotiation.cc:407] Connection header received
0929 19:00:52.969313 (+ 543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:00:52.969315 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:00:52.969329 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:00:52.969411 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:00:52.970067 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:52.970888 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:52.971911 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:52.972110 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:52.972185 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:52.975150 (+ 2965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:00:52.975151 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:00:52.975151 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:00:52.975159 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:00:52.976678 (+ 1519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:00:52.977084 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:00:52.977085 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:00:52.977085 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:00:52.977136 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:00:52.977370 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:00:52.977371 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:00:52.977371 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:00:52.977484 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:00:52.977562 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 19:00:52.977854 (+ 292us) server_negotiation.cc:299] Negotiation successful
0929 19:00:52.977938 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 19:00:52.978174 1600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:52.968462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39367 (local address 127.0.98.1:50803)
0929 19:00:52.968631 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0929 19:00:52.968669 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 19:00:52.969194 (+ 525us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:00:52.969492 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:00:52.969497 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:00:52.969775 (+ 278us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:00:52.969950 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:00:52.969954 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:00:52.970988 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:00:52.970989 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:00:52.971790 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:00:52.971792 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:00:52.972190 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:00:52.972191 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:00:52.972326 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:52.972847 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:00:52.972856 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:00:52.974953 (+ 2097us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:00:52.976759 (+ 1806us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:00:52.976764 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:00:52.976766 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:00:52.977023 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:00:52.977184 (+ 161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:00:52.977184 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:00:52.977185 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:00:52.977261 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:00:52.977552 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:00:52.977557 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:00:52.977702 (+ 145us) client_negotiation.cc:769] Sending connection context
0929 19:00:52.977803 (+ 101us) client_negotiation.cc:240] Negotiation successful
0929 19:00:52.977949 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":86,"threads_started":1}
I20260929 19:00:52.979651 1707 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39367
I20260929 19:00:52.979732 1707 heartbeater.cc:464] Registering TS with master...
I20260929 19:00:52.979888 1707 heartbeater.cc:511] Master 127.0.98.62:39367 requested a full tablet report, sending...
I20260929 19:00:52.980386 1503 ts_manager.cc:194] Registered new tserver with Master: ddad36ae0ddf437f9f8c6f86a538b26a (127.0.98.1:34477)
I20260929 19:00:52.981215 1503 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:50803
I20260929 19:00:52.986295 1572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:52.979654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:32813 (local address 127.0.98.62:39367)
0929 19:00:52.979691 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0929 19:00:52.979693 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:00:52.980191 (+ 498us) server_negotiation.cc:407] Connection header received
0929 19:00:52.980254 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:00:52.980255 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:00:52.980264 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:00:52.980312 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:00:52.980670 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:52.981200 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:52.982824 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:52.982990 (+ 166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:52.983065 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:52.984006 (+ 941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:00:52.984007 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:00:52.984008 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:00:52.984014 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:00:52.985290 (+ 1276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:00:52.985658 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:00:52.985659 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:00:52.985659 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:00:52.985702 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:00:52.985903 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:00:52.985904 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:00:52.985904 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:00:52.985982 (+ 78us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:00:52.986031 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0929 19:00:52.986196 (+ 165us) server_negotiation.cc:299] Negotiation successful
0929 19:00:52.986245 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260929 19:00:52.986397 1713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:52.979614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39367 (local address 127.0.98.1:32813)
0929 19:00:52.980143 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0929 19:00:52.980153 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 19:00:52.980204 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:00:52.980313 (+ 109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:00:52.980313 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:00:52.980526 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:00:52.980580 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:00:52.980581 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:00:52.981292 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:00:52.981293 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:00:52.982747 (+ 1454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:00:52.982750 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:00:52.983088 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:00:52.983089 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:00:52.983161 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:52.983509 (+ 348us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:00:52.983515 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:00:52.983918 (+ 403us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:00:52.985368 (+ 1450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:00:52.985369 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:00:52.985369 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:00:52.985579 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:00:52.985763 (+ 184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:00:52.985764 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:00:52.985764 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:00:52.985831 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:00:52.986049 (+ 218us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:00:52.986050 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:00:52.986095 (+ 45us) client_negotiation.cc:769] Sending connection context
0929 19:00:52.986176 (+ 81us) client_negotiation.cc:240] Negotiation successful
0929 19:00:52.986271 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":64,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708447806542-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708447806542-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260929 19:00:53.001204 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUbnLHE/build/release/bin/kudu
/tmp/dist-test-taskUbnLHE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708447806542-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708447806542-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708447806542-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708447806542-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39367
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708447806542-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708447806542-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708447806542-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708447806542-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:00:53.078593 1716 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:00:53.078795 1716 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:00:53.078835 1716 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:00:53.080018 1716 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:00:53.080266 1716 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:00:53.080330 1716 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 19:00:53.080372 1716 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 19:00:53.082050 1716 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708447806542-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708447806542-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708447806542-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708447806542-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:39367
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1716
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708447806542-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 18:09:13 UTC on ccycloud.quasar-hpkalf.root.comops.site
build id 4680913
I20260929 19:00:53.082299 1716 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:00:53.082511 1716 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:00:53.087236 1721 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:00:53.087234 1722 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:00:53.087236 1724 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:00:53.087253 1716 server_base.cc:1034] running on GCE node
I20260929 19:00:53.087586 1716 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:00:53.087771 1716 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:00:53.088935 1716 hybrid_clock.cc:648] HybridClock initialized: now 1790708453088913 us; error 33 us; skew 500 ppm
Sep 29 19:00:53 dist-test-slave-lqkv krb5kdc[1459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708453, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:00:53.091235 1716 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 19:00:53.091640 1716 webserver.cc:460] Webserver started at http://127.0.98.2:40979/ using document root <none> and password file <none>
I20260929 19:00:53.091853 1716 fs_manager.cc:359] Metadata directory not provided
I20260929 19:00:53.091917 1716 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:00:53.092058 1716 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:00:53.093062 1716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708447806542-392-0/minicluster-data/ts-1/data/instance:
uuid: "85ac5a425d1d44d793797f6f5e25dae5"
format_stamp: "Formatted at 2026-09-29 19:00:53 on dist-test-slave-lqkv"
server_key: "04391e50c7c8e6e95414f9297b8587b9"
server_key_iv: "036df49e1fe8c40da3238fdf03ef9d24"
server_key_version: "encryptionkey@0"
I20260929 19:00:53.093385 1716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708447806542-392-0/minicluster-data/ts-1/wal/instance:
uuid: "85ac5a425d1d44d793797f6f5e25dae5"
format_stamp: "Formatted at 2026-09-29 19:00:53 on dist-test-slave-lqkv"
server_key: "04391e50c7c8e6e95414f9297b8587b9"
server_key_iv: "036df49e1fe8c40da3238fdf03ef9d24"
server_key_version: "encryptionkey@0"
I20260929 19:00:53.094606 1716 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 19:00:53.095254 1731 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:00:53.095398 1716 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 19:00:53.095482 1716 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708447806542-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708447806542-392-0/minicluster-data/ts-1/wal
uuid: "85ac5a425d1d44d793797f6f5e25dae5"
format_stamp: "Formatted at 2026-09-29 19:00:53 on dist-test-slave-lqkv"
server_key: "04391e50c7c8e6e95414f9297b8587b9"
server_key_iv: "036df49e1fe8c40da3238fdf03ef9d24"
server_key_version: "encryptionkey@0"
I20260929 19:00:53.095558 1716 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708447806542-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708447806542-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708447806542-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:00:53.111497 1716 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:00:53.113166 1716 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:00:53.113318 1716 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:00:53.113760 1716 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:00:53.113821 1716 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:00:53.113862 1716 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:00:53.113893 1716 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 19:00:53 dist-test-slave-lqkv krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708453, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 19:00:53.120715 1716 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42011
I20260929 19:00:53.120792 1846 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42011 every 8 connection(s)
I20260929 19:00:53.121086 1716 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708447806542-392-0/minicluster-data/ts-1/data/info.pb
I20260929 19:00:53.123359 1572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:53.114214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37117 (local address 127.0.98.62:39367)
0929 19:00:53.114254 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0929 19:00:53.114257 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 19:00:53.114452 (+ 195us) server_negotiation.cc:407] Connection header received
0929 19:00:53.114948 (+ 496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:00:53.114950 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:00:53.114962 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:00:53.115029 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:00:53.115733 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:53.116502 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:53.118236 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:53.118431 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:53.118522 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:53.120958 (+ 2436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:00:53.120960 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:00:53.120960 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:00:53.120975 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:00:53.122335 (+ 1360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:00:53.122669 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:00:53.122669 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:00:53.122670 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:00:53.122715 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:00:53.122887 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:00:53.122888 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:00:53.122888 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:00:53.122970 (+ 82us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:00:53.123022 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0929 19:00:53.123240 (+ 218us) server_negotiation.cc:299] Negotiation successful
0929 19:00:53.123301 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260929 19:00:53.123512 1740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:53.114207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39367 (local address 127.0.98.2:37117)
0929 19:00:53.114351 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0929 19:00:53.114380 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0929 19:00:53.114837 (+ 457us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:00:53.115093 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:00:53.115097 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:00:53.115381 (+ 284us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:00:53.115652 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:00:53.115657 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:00:53.116636 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:00:53.116637 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:00:53.118137 (+ 1500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:00:53.118140 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:00:53.118543 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:00:53.118544 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:00:53.118706 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:53.119176 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:00:53.119187 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:00:53.120862 (+ 1675us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:00:53.122405 (+ 1543us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:00:53.122410 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:00:53.122411 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:00:53.122607 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:00:53.122754 (+ 147us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:00:53.122754 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:00:53.122754 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:00:53.122830 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:00:53.123023 (+ 193us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:00:53.123027 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:00:53.123138 (+ 111us) client_negotiation.cc:769] Sending connection context
0929 19:00:53.123214 (+ 76us) client_negotiation.cc:240] Negotiation successful
0929 19:00:53.123317 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":74,"threads_started":1}
I20260929 19:00:53.123968 1847 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39367
I20260929 19:00:53.124066 1847 heartbeater.cc:464] Registering TS with master...
I20260929 19:00:53.124225 1847 heartbeater.cc:511] Master 127.0.98.62:39367 requested a full tablet report, sending...
I20260929 19:00:53.124589 1504 ts_manager.cc:194] Registered new tserver with Master: 85ac5a425d1d44d793797f6f5e25dae5 (127.0.98.2:42011)
I20260929 19:00:53.125165 1504 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:37117
I20260929 19:00:53.125417 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUbnLHE/build/release/bin/kudu as pid 1716
I20260929 19:00:53.125509 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708447806542-392-0/minicluster-data/ts-1/wal/instance
I20260929 19:00:53.125687 392 external_mini_cluster.cc:1428] Setting key 2e13347aede2ccc37e3ed30351afad93
I20260929 19:00:53.130803 1572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:53.125047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50509 (local address 127.0.98.62:39367)
0929 19:00:53.125083 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0929 19:00:53.125085 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:00:53.125129 (+ 44us) server_negotiation.cc:407] Connection header received
0929 19:00:53.125193 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:00:53.125194 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:00:53.125201 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:00:53.125267 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:00:53.125621 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:53.126347 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:53.126983 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:53.127318 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:53.127387 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:53.128341 (+ 954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:00:53.128342 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:00:53.128342 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:00:53.128347 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:00:53.129716 (+ 1369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:00:53.130108 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:00:53.130109 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:00:53.130109 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:00:53.130161 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:00:53.130353 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:00:53.130354 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:00:53.130354 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:00:53.130469 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:00:53.130517 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0929 19:00:53.130685 (+ 168us) server_negotiation.cc:299] Negotiation successful
0929 19:00:53.130737 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260929 19:00:53.130982 1853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:53.124948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39367 (local address 127.0.98.2:50509)
0929 19:00:53.125074 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0929 19:00:53.125085 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 19:00:53.125142 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:00:53.125272 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:00:53.125273 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:00:53.125445 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:00:53.125517 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:00:53.125519 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:00:53.126425 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:00:53.126426 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:00:53.126903 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:00:53.126905 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:00:53.127390 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:00:53.127391 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:00:53.127434 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:53.127868 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:00:53.127873 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:00:53.128258 (+ 385us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:00:53.129808 (+ 1550us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:00:53.129809 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:00:53.129810 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:00:53.130035 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:00:53.130227 (+ 192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:00:53.130228 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:00:53.130228 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:00:53.130292 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:00:53.130537 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:00:53.130538 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:00:53.130586 (+ 48us) client_negotiation.cc:769] Sending connection context
0929 19:00:53.130672 (+ 86us) client_negotiation.cc:240] Negotiation successful
0929 19:00:53.130792 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":79,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708447806542-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708447806542-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260929 19:00:53.149137 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUbnLHE/build/release/bin/kudu
/tmp/dist-test-taskUbnLHE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708447806542-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708447806542-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708447806542-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708447806542-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39367
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708447806542-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708447806542-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708447806542-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708447806542-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:00:53.226342 1856 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:00:53.226512 1856 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:00:53.226534 1856 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:00:53.227615 1856 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:00:53.227653 1856 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:00:53.227674 1856 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 19:00:53.227725 1856 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 19:00:53.229177 1856 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708447806542-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708447806542-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708447806542-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708447806542-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:39367
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1856
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708447806542-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 18:09:13 UTC on ccycloud.quasar-hpkalf.root.comops.site
build id 4680913
I20260929 19:00:53.229393 1856 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:00:53.229600 1856 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:00:53.234241 1861 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:00:53.234251 1864 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:00:53.234292 1862 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:00:53.234452 1856 server_base.cc:1034] running on GCE node
I20260929 19:00:53.234694 1856 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:00:53.234886 1856 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:00:53.236033 1856 hybrid_clock.cc:648] HybridClock initialized: now 1790708453236029 us; error 33 us; skew 500 ppm
Sep 29 19:00:53 dist-test-slave-lqkv krb5kdc[1459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708453, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:00:53.238427 1856 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 19:00:53.238837 1856 webserver.cc:460] Webserver started at http://127.0.98.3:46847/ using document root <none> and password file <none>
I20260929 19:00:53.239063 1856 fs_manager.cc:359] Metadata directory not provided
I20260929 19:00:53.239105 1856 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:00:53.239243 1856 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:00:53.240211 1856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708447806542-392-0/minicluster-data/ts-2/data/instance:
uuid: "a4b5b67825d8469fa032a2b34f32da06"
format_stamp: "Formatted at 2026-09-29 19:00:53 on dist-test-slave-lqkv"
server_key: "6957247a2b174e5871b742ff9c92b8bd"
server_key_iv: "0994b78f0bb590c2ee4608ac119b4568"
server_key_version: "encryptionkey@0"
I20260929 19:00:53.240563 1856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708447806542-392-0/minicluster-data/ts-2/wal/instance:
uuid: "a4b5b67825d8469fa032a2b34f32da06"
format_stamp: "Formatted at 2026-09-29 19:00:53 on dist-test-slave-lqkv"
server_key: "6957247a2b174e5871b742ff9c92b8bd"
server_key_iv: "0994b78f0bb590c2ee4608ac119b4568"
server_key_version: "encryptionkey@0"
I20260929 19:00:53.241871 1856 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 19:00:53.242571 1871 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:00:53.242753 1856 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 19:00:53.242854 1856 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708447806542-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708447806542-392-0/minicluster-data/ts-2/wal
uuid: "a4b5b67825d8469fa032a2b34f32da06"
format_stamp: "Formatted at 2026-09-29 19:00:53 on dist-test-slave-lqkv"
server_key: "6957247a2b174e5871b742ff9c92b8bd"
server_key_iv: "0994b78f0bb590c2ee4608ac119b4568"
server_key_version: "encryptionkey@0"
I20260929 19:00:53.242920 1856 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708447806542-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708447806542-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708447806542-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:00:53.262454 1856 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:00:53.264241 1856 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:00:53.264384 1856 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:00:53.264890 1856 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:00:53.264932 1856 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:00:53.264954 1856 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:00:53.264967 1856 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 19:00:53 dist-test-slave-lqkv krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708453, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 19:00:53.271924 1856 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40465
I20260929 19:00:53.271989 1986 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40465 every 8 connection(s)
I20260929 19:00:53.272277 1856 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708447806542-392-0/minicluster-data/ts-2/data/info.pb
I20260929 19:00:53.273168 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUbnLHE/build/release/bin/kudu as pid 1856
I20260929 19:00:53.273242 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708447806542-392-0/minicluster-data/ts-2/wal/instance
I20260929 19:00:53.273423 392 external_mini_cluster.cc:1428] Setting key 437d0e50013d64725b9d68d5b6b89297
I20260929 19:00:53.274765 1572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:53.265362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55045 (local address 127.0.98.62:39367)
0929 19:00:53.265432 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0929 19:00:53.265435 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 19:00:53.265598 (+ 163us) server_negotiation.cc:407] Connection header received
0929 19:00:53.266046 (+ 448us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:00:53.266048 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:00:53.266060 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:00:53.266133 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:00:53.266734 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:53.267552 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:53.269200 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:53.269502 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:53.269587 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:53.271755 (+ 2168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:00:53.271756 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:00:53.271757 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:00:53.271764 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:00:53.273398 (+ 1634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:00:53.273824 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:00:53.273826 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:00:53.273826 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:00:53.273903 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:00:53.274145 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:00:53.274146 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:00:53.274146 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:00:53.274282 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:00:53.274345 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0929 19:00:53.274599 (+ 254us) server_negotiation.cc:299] Negotiation successful
0929 19:00:53.274675 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260929 19:00:53.274940 1880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:53.265326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39367 (local address 127.0.98.3:55045)
0929 19:00:53.265475 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0929 19:00:53.265513 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 19:00:53.265955 (+ 442us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:00:53.266204 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:00:53.266207 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:00:53.266425 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:00:53.266665 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:00:53.266670 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:00:53.267658 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:00:53.267660 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:00:53.269116 (+ 1456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:00:53.269118 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:00:53.269605 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:00:53.269606 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:00:53.269735 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:53.270178 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:00:53.270187 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:00:53.271643 (+ 1456us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:00:53.273470 (+ 1827us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:00:53.273476 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:00:53.273477 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:00:53.273734 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:00:53.273960 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:00:53.273961 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:00:53.273961 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:00:53.274074 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:00:53.274336 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:00:53.274341 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:00:53.274470 (+ 129us) client_negotiation.cc:769] Sending connection context
0929 19:00:53.274577 (+ 107us) client_negotiation.cc:240] Negotiation successful
0929 19:00:53.274725 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":74,"threads_started":1}
I20260929 19:00:53.275424 1987 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39367
I20260929 19:00:53.275507 1987 heartbeater.cc:464] Registering TS with master...
I20260929 19:00:53.275697 1987 heartbeater.cc:511] Master 127.0.98.62:39367 requested a full tablet report, sending...
I20260929 19:00:53.276053 1504 ts_manager.cc:194] Registered new tserver with Master: a4b5b67825d8469fa032a2b34f32da06 (127.0.98.3:40465)
I20260929 19:00:53.276513 1504 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:55045
I20260929 19:00:53.281528 1572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:53.276353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35073 (local address 127.0.98.62:39367)
0929 19:00:53.276384 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0929 19:00:53.276386 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:00:53.276822 (+ 436us) server_negotiation.cc:407] Connection header received
0929 19:00:53.276844 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:00:53.276845 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:00:53.276854 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:00:53.276910 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:00:53.277261 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:53.277708 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:53.278396 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:53.278605 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:53.278676 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:53.279396 (+ 720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:00:53.279397 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:00:53.279397 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:00:53.279404 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:00:53.280446 (+ 1042us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:00:53.280774 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:00:53.280775 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:00:53.280775 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:00:53.280850 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:00:53.281053 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:00:53.281054 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:00:53.281055 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:00:53.281158 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:00:53.281230 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 19:00:53.281374 (+ 144us) server_negotiation.cc:299] Negotiation successful
0929 19:00:53.281436 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260929 19:00:53.281539 1993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:53.276283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39367 (local address 127.0.98.3:35073)
0929 19:00:53.276689 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0929 19:00:53.276700 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 19:00:53.276776 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:00:53.276939 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:00:53.276940 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:00:53.277115 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:00:53.277176 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:00:53.277177 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:00:53.277794 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:00:53.277794 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:00:53.278302 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:00:53.278304 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:00:53.278674 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:00:53.278675 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:00:53.278742 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:53.279074 (+ 332us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:00:53.279078 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:00:53.279312 (+ 234us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:00:53.280546 (+ 1234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:00:53.280547 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:00:53.280548 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:00:53.280705 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:00:53.280922 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:00:53.280923 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:00:53.280923 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:00:53.280984 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:00:53.281221 (+ 237us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:00:53.281222 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:00:53.281262 (+ 40us) client_negotiation.cc:769] Sending connection context
0929 19:00:53.281348 (+ 86us) client_negotiation.cc:240] Negotiation successful
0929 19:00:53.281437 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":363,"mutex_wait_us":6,"thread_start_us":74,"threads_started":1}
I20260929 19:00:53.285015 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 19:00:53 dist-test-slave-lqkv krb5kdc[1459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708453, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 19:00:53 dist-test-slave-lqkv krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708453, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 19:00:53.313953 1572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:53.307167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44554 (local address 127.0.98.62:39367)
0929 19:00:53.307210 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0929 19:00:53.307211 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 19:00:53.307218 (+ 7us) server_negotiation.cc:407] Connection header received
0929 19:00:53.307240 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:00:53.307241 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:00:53.307249 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:00:53.307333 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:00:53.307862 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:53.308363 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:53.309230 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:53.309406 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:53.309458 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:53.311458 (+ 2000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:00:53.311459 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:00:53.311460 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:00:53.311467 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:00:53.312773 (+ 1306us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:00:53.313171 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:00:53.313172 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:00:53.313173 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:00:53.313227 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:00:53.313419 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:00:53.313420 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:00:53.313420 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:00:53.313530 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:00:53.313626 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0929 19:00:53.313779 (+ 153us) server_negotiation.cc:299] Negotiation successful
0929 19:00:53.313849 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260929 19:00:53.315380 1504 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44554:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 19:00:53.316869 1504 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 19:00:53.322780 2005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:53.318883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42011 (local address 127.0.0.1:55752)
0929 19:00:53.319032 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0929 19:00:53.319072 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 19:00:53.319139 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:00:53.319459 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:00:53.319462 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:00:53.319464 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:00:53.319531 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:00:53.319533 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:00:53.320523 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:00:53.320524 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:00:53.321609 (+ 1085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:00:53.321612 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:00:53.322500 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:00:53.322501 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:00:53.322586 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:53.322600 (+ 14us) client_negotiation.cc:769] Sending connection context
0929 19:00:53.322671 (+ 71us) client_negotiation.cc:240] Negotiation successful
0929 19:00:53.322714 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":49,"threads_started":1}
I20260929 19:00:53.323065 2008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:53.319152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55752 (local address 127.0.98.2:42011)
0929 19:00:53.319284 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0929 19:00:53.319289 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 19:00:53.319299 (+ 10us) server_negotiation.cc:407] Connection header received
0929 19:00:53.319337 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:00:53.319339 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:00:53.319355 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:00:53.319440 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:00:53.319643 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:53.320438 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:53.321699 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:53.322412 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:53.322488 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:53.322560 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 19:00:53.322696 (+ 136us) server_negotiation.cc:299] Negotiation successful
0929 19:00:53.322745 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":74,"threads_started":1}
I20260929 19:00:53.323084 2007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:53.318892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40465 (local address 127.0.0.1:48198)
0929 19:00:53.319187 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0929 19:00:53.319198 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 19:00:53.319273 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:00:53.319516 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:00:53.319517 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:00:53.319519 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:00:53.319556 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:00:53.319557 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:00:53.320529 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:00:53.320530 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:00:53.321885 (+ 1355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:00:53.321887 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:00:53.322842 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:00:53.322844 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:00:53.322918 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:53.322922 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 19:00:53.322969 (+ 47us) client_negotiation.cc:240] Negotiation successful
0929 19:00:53.323017 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"mutex_wait_us":29,"thread_start_us":71,"threads_started":1}
I20260929 19:00:53.323146 2006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:53.318976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48198 (local address 127.0.98.3:40465)
0929 19:00:53.319136 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0929 19:00:53.319141 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 19:00:53.319248 (+ 107us) server_negotiation.cc:407] Connection header received
0929 19:00:53.319361 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:00:53.319364 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:00:53.319377 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:00:53.319458 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:00:53.319651 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:53.320440 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:53.321971 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:53.322673 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:53.322740 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:53.322809 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0929 19:00:53.323014 (+ 205us) server_negotiation.cc:299] Negotiation successful
0929 19:00:53.323053 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":72,"threads_started":1}
I20260929 19:00:53.323733 1921 tablet_service.cc:1467] Processing CreateTablet for tablet c95af5937e8046cfb96b47695df7cb15 (DEFAULT_TABLE table=test-table [id=dc4d1874e5ab4873a3068cff7124f23d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:00:53.324019 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:53.319335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36098 (local address 127.0.98.1:34477)
0929 19:00:53.320597 (+ 1262us) server_negotiation.cc:206] Beginning negotiation
0929 19:00:53.320601 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 19:00:53.320610 (+ 9us) server_negotiation.cc:407] Connection header received
0929 19:00:53.320642 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:00:53.320644 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:00:53.320657 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:00:53.320732 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:00:53.320885 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:53.321932 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:53.322918 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:53.323636 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:53.323705 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:53.323765 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0929 19:00:53.323884 (+ 119us) server_negotiation.cc:299] Negotiation successful
0929 19:00:53.323919 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1229,"thread_start_us":84,"threads_started":1}
I20260929 19:00:53.323966 1781 tablet_service.cc:1467] Processing CreateTablet for tablet c95af5937e8046cfb96b47695df7cb15 (DEFAULT_TABLE table=test-table [id=dc4d1874e5ab4873a3068cff7124f23d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:00:53.324115 1921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c95af5937e8046cfb96b47695df7cb15. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:00:53.324051 2004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:53.319074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34477 (local address 127.0.0.1:36098)
0929 19:00:53.319503 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0929 19:00:53.319513 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 19:00:53.319556 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:00:53.320754 (+ 1198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:00:53.320755 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:00:53.320757 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:00:53.320791 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:00:53.320794 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:00:53.322015 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:00:53.322015 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:00:53.322830 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:00:53.322832 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:00:53.323719 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:00:53.323720 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:00:53.323780 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:53.323785 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 19:00:53.323861 (+ 76us) client_negotiation.cc:240] Negotiation successful
0929 19:00:53.323901 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":63,"threads_started":1}
I20260929 19:00:53.324273 1781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c95af5937e8046cfb96b47695df7cb15. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:00:53.325060 1641 tablet_service.cc:1467] Processing CreateTablet for tablet c95af5937e8046cfb96b47695df7cb15 (DEFAULT_TABLE table=test-table [id=dc4d1874e5ab4873a3068cff7124f23d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:00:53.325366 1641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c95af5937e8046cfb96b47695df7cb15. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:00:53.327104 2010 tablet_bootstrap.cc:492] T c95af5937e8046cfb96b47695df7cb15 P a4b5b67825d8469fa032a2b34f32da06: Bootstrap starting.
I20260929 19:00:53.327744 2010 tablet_bootstrap.cc:654] T c95af5937e8046cfb96b47695df7cb15 P a4b5b67825d8469fa032a2b34f32da06: Neither blocks nor log segments found. Creating new log.
I20260929 19:00:53.328030 2011 tablet_bootstrap.cc:492] T c95af5937e8046cfb96b47695df7cb15 P 85ac5a425d1d44d793797f6f5e25dae5: Bootstrap starting.
I20260929 19:00:53.328110 2010 log.cc:824] T c95af5937e8046cfb96b47695df7cb15 P a4b5b67825d8469fa032a2b34f32da06: Log is configured to *not* fsync() on all Append() calls
I20260929 19:00:53.328699 2011 tablet_bootstrap.cc:654] T c95af5937e8046cfb96b47695df7cb15 P 85ac5a425d1d44d793797f6f5e25dae5: Neither blocks nor log segments found. Creating new log.
I20260929 19:00:53.328900 2010 tablet_bootstrap.cc:492] T c95af5937e8046cfb96b47695df7cb15 P a4b5b67825d8469fa032a2b34f32da06: No bootstrap required, opened a new log
I20260929 19:00:53.328974 2010 ts_tablet_manager.cc:1397] T c95af5937e8046cfb96b47695df7cb15 P a4b5b67825d8469fa032a2b34f32da06: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 19:00:53.329087 2011 log.cc:824] T c95af5937e8046cfb96b47695df7cb15 P 85ac5a425d1d44d793797f6f5e25dae5: Log is configured to *not* fsync() on all Append() calls
I20260929 19:00:53.329190 2013 tablet_bootstrap.cc:492] T c95af5937e8046cfb96b47695df7cb15 P ddad36ae0ddf437f9f8c6f86a538b26a: Bootstrap starting.
I20260929 19:00:53.329650 2013 tablet_bootstrap.cc:654] T c95af5937e8046cfb96b47695df7cb15 P ddad36ae0ddf437f9f8c6f86a538b26a: Neither blocks nor log segments found. Creating new log.
I20260929 19:00:53.329918 2013 log.cc:824] T c95af5937e8046cfb96b47695df7cb15 P ddad36ae0ddf437f9f8c6f86a538b26a: Log is configured to *not* fsync() on all Append() calls
I20260929 19:00:53.330560 2011 tablet_bootstrap.cc:492] T c95af5937e8046cfb96b47695df7cb15 P 85ac5a425d1d44d793797f6f5e25dae5: No bootstrap required, opened a new log
I20260929 19:00:53.330636 2011 ts_tablet_manager.cc:1397] T c95af5937e8046cfb96b47695df7cb15 P 85ac5a425d1d44d793797f6f5e25dae5: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260929 19:00:53.330709 2013 tablet_bootstrap.cc:492] T c95af5937e8046cfb96b47695df7cb15 P ddad36ae0ddf437f9f8c6f86a538b26a: No bootstrap required, opened a new log
I20260929 19:00:53.330777 2013 ts_tablet_manager.cc:1397] T c95af5937e8046cfb96b47695df7cb15 P ddad36ae0ddf437f9f8c6f86a538b26a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 19:00:53.330715 2010 raft_consensus.cc:348] T c95af5937e8046cfb96b47695df7cb15 P a4b5b67825d8469fa032a2b34f32da06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4b5b67825d8469fa032a2b34f32da06" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40465 } } peers { permanent_uuid: "85ac5a425d1d44d793797f6f5e25dae5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42011 } } peers { permanent_uuid: "ddad36ae0ddf437f9f8c6f86a538b26a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34477 } }
I20260929 19:00:53.330871 2010 raft_consensus.cc:374] T c95af5937e8046cfb96b47695df7cb15 P a4b5b67825d8469fa032a2b34f32da06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:00:53.330909 2010 raft_consensus.cc:729] T c95af5937e8046cfb96b47695df7cb15 P a4b5b67825d8469fa032a2b34f32da06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4b5b67825d8469fa032a2b34f32da06, State: Initialized, Role: FOLLOWER
I20260929 19:00:53.331025 2010 consensus_queue.cc:260] T c95af5937e8046cfb96b47695df7cb15 P a4b5b67825d8469fa032a2b34f32da06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4b5b67825d8469fa032a2b34f32da06" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40465 } } peers { permanent_uuid: "85ac5a425d1d44d793797f6f5e25dae5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42011 } } peers { permanent_uuid: "ddad36ae0ddf437f9f8c6f86a538b26a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34477 } }
I20260929 19:00:53.331322 2010 ts_tablet_manager.cc:1428] T c95af5937e8046cfb96b47695df7cb15 P a4b5b67825d8469fa032a2b34f32da06: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 19:00:53.331434 1987 heartbeater.cc:503] Master 127.0.98.62:39367 was elected leader, sending a full tablet report...
I20260929 19:00:53.332579 2013 raft_consensus.cc:348] T c95af5937e8046cfb96b47695df7cb15 P ddad36ae0ddf437f9f8c6f86a538b26a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4b5b67825d8469fa032a2b34f32da06" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40465 } } peers { permanent_uuid: "85ac5a425d1d44d793797f6f5e25dae5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42011 } } peers { permanent_uuid: "ddad36ae0ddf437f9f8c6f86a538b26a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34477 } }
I20260929 19:00:53.332701 2013 raft_consensus.cc:374] T c95af5937e8046cfb96b47695df7cb15 P ddad36ae0ddf437f9f8c6f86a538b26a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:00:53.332594 2011 raft_consensus.cc:348] T c95af5937e8046cfb96b47695df7cb15 P 85ac5a425d1d44d793797f6f5e25dae5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4b5b67825d8469fa032a2b34f32da06" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40465 } } peers { permanent_uuid: "85ac5a425d1d44d793797f6f5e25dae5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42011 } } peers { permanent_uuid: "ddad36ae0ddf437f9f8c6f86a538b26a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34477 } }
I20260929 19:00:53.332720 2013 raft_consensus.cc:729] T c95af5937e8046cfb96b47695df7cb15 P ddad36ae0ddf437f9f8c6f86a538b26a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddad36ae0ddf437f9f8c6f86a538b26a, State: Initialized, Role: FOLLOWER
I20260929 19:00:53.332727 2011 raft_consensus.cc:374] T c95af5937e8046cfb96b47695df7cb15 P 85ac5a425d1d44d793797f6f5e25dae5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:00:53.332764 2011 raft_consensus.cc:729] T c95af5937e8046cfb96b47695df7cb15 P 85ac5a425d1d44d793797f6f5e25dae5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85ac5a425d1d44d793797f6f5e25dae5, State: Initialized, Role: FOLLOWER
I20260929 19:00:53.332813 2013 consensus_queue.cc:260] T c95af5937e8046cfb96b47695df7cb15 P ddad36ae0ddf437f9f8c6f86a538b26a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4b5b67825d8469fa032a2b34f32da06" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40465 } } peers { permanent_uuid: "85ac5a425d1d44d793797f6f5e25dae5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42011 } } peers { permanent_uuid: "ddad36ae0ddf437f9f8c6f86a538b26a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34477 } }
I20260929 19:00:53.332880 2011 consensus_queue.cc:260] T c95af5937e8046cfb96b47695df7cb15 P 85ac5a425d1d44d793797f6f5e25dae5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4b5b67825d8469fa032a2b34f32da06" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40465 } } peers { permanent_uuid: "85ac5a425d1d44d793797f6f5e25dae5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42011 } } peers { permanent_uuid: "ddad36ae0ddf437f9f8c6f86a538b26a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34477 } }
I20260929 19:00:53.333096 1707 heartbeater.cc:503] Master 127.0.98.62:39367 was elected leader, sending a full tablet report...
I20260929 19:00:53.333143 2011 ts_tablet_manager.cc:1428] T c95af5937e8046cfb96b47695df7cb15 P 85ac5a425d1d44d793797f6f5e25dae5: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260929 19:00:53.333205 1847 heartbeater.cc:503] Master 127.0.98.62:39367 was elected leader, sending a full tablet report...
I20260929 19:00:53.333333 2013 ts_tablet_manager.cc:1428] T c95af5937e8046cfb96b47695df7cb15 P ddad36ae0ddf437f9f8c6f86a538b26a: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
W20260929 19:00:53.371775 1848 tablet.cc:2369] T c95af5937e8046cfb96b47695df7cb15 P 85ac5a425d1d44d793797f6f5e25dae5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 19:00:53.476367 1708 tablet.cc:2369] T c95af5937e8046cfb96b47695df7cb15 P ddad36ae0ddf437f9f8c6f86a538b26a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 19:00:53.522802 1988 tablet.cc:2369] T c95af5937e8046cfb96b47695df7cb15 P a4b5b67825d8469fa032a2b34f32da06: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 19:00:53.594398 2016 raft_consensus.cc:482] T c95af5937e8046cfb96b47695df7cb15 P a4b5b67825d8469fa032a2b34f32da06 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 19:00:53.594558 2016 raft_consensus.cc:504] T c95af5937e8046cfb96b47695df7cb15 P a4b5b67825d8469fa032a2b34f32da06 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4b5b67825d8469fa032a2b34f32da06" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40465 } } peers { permanent_uuid: "85ac5a425d1d44d793797f6f5e25dae5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42011 } } peers { permanent_uuid: "ddad36ae0ddf437f9f8c6f86a538b26a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34477 } }
I20260929 19:00:53.594918 2016 leader_election.cc:288] T c95af5937e8046cfb96b47695df7cb15 P a4b5b67825d8469fa032a2b34f32da06 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85ac5a425d1d44d793797f6f5e25dae5 (127.0.98.2:42011), ddad36ae0ddf437f9f8c6f86a538b26a (127.0.98.1:34477)
I20260929 19:00:53.598948 2008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:53.595243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38759 (local address 127.0.98.2:42011)
0929 19:00:53.595369 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0929 19:00:53.595371 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:00:53.595381 (+ 10us) server_negotiation.cc:407] Connection header received
0929 19:00:53.595410 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:00:53.595411 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:00:53.595422 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:00:53.595511 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:00:53.595677 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:53.596432 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:53.598012 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:53.598572 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:53.598652 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:53.598664 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0929 19:00:53.598844 (+ 180us) server_negotiation.cc:299] Negotiation successful
0929 19:00:53.598880 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260929 19:00:53.598948 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:53.595383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54435 (local address 127.0.98.1:34477)
0929 19:00:53.595426 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0929 19:00:53.595428 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:00:53.595541 (+ 113us) server_negotiation.cc:407] Connection header received
0929 19:00:53.595568 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:00:53.595569 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:00:53.595587 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:00:53.595634 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:00:53.595781 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:53.596450 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:53.598012 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:53.598638 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:53.598715 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:53.598725 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0929 19:00:53.598844 (+ 119us) server_negotiation.cc:299] Negotiation successful
0929 19:00:53.598880 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260929 19:00:53.598959 2019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:53.595293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34477 (local address 127.0.98.3:54435)
0929 19:00:53.595433 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0929 19:00:53.595441 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0929 19:00:53.595507 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:00:53.595651 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:00:53.595652 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:00:53.595654 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:00:53.595713 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:00:53.595715 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:00:53.596522 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:00:53.596522 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:00:53.597924 (+ 1402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:00:53.597926 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:00:53.598718 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:00:53.598719 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:00:53.598776 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:53.598780 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 19:00:53.598831 (+ 51us) client_negotiation.cc:240] Negotiation successful
0929 19:00:53.598861 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"mutex_wait_us":95,"thread_start_us":89,"threads_started":1}
I20260929 19:00:53.598978 1880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:53.595164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42011 (local address 127.0.98.3:38759)
0929 19:00:53.595243 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0929 19:00:53.595257 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 19:00:53.595318 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:00:53.595514 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:00:53.595515 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:00:53.595516 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:00:53.595589 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:00:53.595591 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:00:53.596522 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:00:53.596522 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:00:53.597924 (+ 1402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:00:53.597926 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:00:53.598663 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:00:53.598679 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:00:53.598758 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:53.598762 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 19:00:53.598825 (+ 63us) client_negotiation.cc:240] Negotiation successful
0929 19:00:53.598861 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58,"mutex_wait_us":74}
I20260929 19:00:53.599324 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c95af5937e8046cfb96b47695df7cb15" candidate_uuid: "a4b5b67825d8469fa032a2b34f32da06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddad36ae0ddf437f9f8c6f86a538b26a" is_pre_election: true
I20260929 19:00:53.599502 1661 raft_consensus.cc:2454] T c95af5937e8046cfb96b47695df7cb15 P ddad36ae0ddf437f9f8c6f86a538b26a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4b5b67825d8469fa032a2b34f32da06 in term 0.
I20260929 19:00:53.599505 1801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c95af5937e8046cfb96b47695df7cb15" candidate_uuid: "a4b5b67825d8469fa032a2b34f32da06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85ac5a425d1d44d793797f6f5e25dae5" is_pre_election: true
I20260929 19:00:53.599615 1801 raft_consensus.cc:2454] T c95af5937e8046cfb96b47695df7cb15 P 85ac5a425d1d44d793797f6f5e25dae5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4b5b67825d8469fa032a2b34f32da06 in term 0.
I20260929 19:00:53.599725 1873 leader_election.cc:302] T c95af5937e8046cfb96b47695df7cb15 P a4b5b67825d8469fa032a2b34f32da06 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a4b5b67825d8469fa032a2b34f32da06, ddad36ae0ddf437f9f8c6f86a538b26a; no voters:
I20260929 19:00:53.599864 2016 raft_consensus.cc:2781] T c95af5937e8046cfb96b47695df7cb15 P a4b5b67825d8469fa032a2b34f32da06 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 19:00:53.599932 2016 raft_consensus.cc:482] T c95af5937e8046cfb96b47695df7cb15 P a4b5b67825d8469fa032a2b34f32da06 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 19:00:53.599954 2016 raft_consensus.cc:3037] T c95af5937e8046cfb96b47695df7cb15 P a4b5b67825d8469fa032a2b34f32da06 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:00:53.600890 2016 raft_consensus.cc:504] T c95af5937e8046cfb96b47695df7cb15 P a4b5b67825d8469fa032a2b34f32da06 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4b5b67825d8469fa032a2b34f32da06" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40465 } } peers { permanent_uuid: "85ac5a425d1d44d793797f6f5e25dae5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42011 } } peers { permanent_uuid: "ddad36ae0ddf437f9f8c6f86a538b26a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34477 } }
I20260929 19:00:53.601018 2016 leader_election.cc:288] T c95af5937e8046cfb96b47695df7cb15 P a4b5b67825d8469fa032a2b34f32da06 [CANDIDATE]: Term 1 election: Requested vote from peers 85ac5a425d1d44d793797f6f5e25dae5 (127.0.98.2:42011), ddad36ae0ddf437f9f8c6f86a538b26a (127.0.98.1:34477)
I20260929 19:00:53.601183 1801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c95af5937e8046cfb96b47695df7cb15" candidate_uuid: "a4b5b67825d8469fa032a2b34f32da06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85ac5a425d1d44d793797f6f5e25dae5"
I20260929 19:00:53.601226 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c95af5937e8046cfb96b47695df7cb15" candidate_uuid: "a4b5b67825d8469fa032a2b34f32da06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddad36ae0ddf437f9f8c6f86a538b26a"
I20260929 19:00:53.601253 1801 raft_consensus.cc:3037] T c95af5937e8046cfb96b47695df7cb15 P 85ac5a425d1d44d793797f6f5e25dae5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:00:53.601323 1661 raft_consensus.cc:3037] T c95af5937e8046cfb96b47695df7cb15 P ddad36ae0ddf437f9f8c6f86a538b26a [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:00:53.601917 1801 raft_consensus.cc:2454] T c95af5937e8046cfb96b47695df7cb15 P 85ac5a425d1d44d793797f6f5e25dae5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4b5b67825d8469fa032a2b34f32da06 in term 1.
I20260929 19:00:53.602006 1661 raft_consensus.cc:2454] T c95af5937e8046cfb96b47695df7cb15 P ddad36ae0ddf437f9f8c6f86a538b26a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4b5b67825d8469fa032a2b34f32da06 in term 1.
I20260929 19:00:53.602129 1873 leader_election.cc:302] T c95af5937e8046cfb96b47695df7cb15 P a4b5b67825d8469fa032a2b34f32da06 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a4b5b67825d8469fa032a2b34f32da06, ddad36ae0ddf437f9f8c6f86a538b26a; no voters:
I20260929 19:00:53.602207 2016 raft_consensus.cc:2781] T c95af5937e8046cfb96b47695df7cb15 P a4b5b67825d8469fa032a2b34f32da06 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:00:53.602350 2016 raft_consensus.cc:686] T c95af5937e8046cfb96b47695df7cb15 P a4b5b67825d8469fa032a2b34f32da06 [term 1 LEADER]: Becoming Leader. State: Replica: a4b5b67825d8469fa032a2b34f32da06, State: Running, Role: LEADER
I20260929 19:00:53.602447 2016 consensus_queue.cc:237] T c95af5937e8046cfb96b47695df7cb15 P a4b5b67825d8469fa032a2b34f32da06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4b5b67825d8469fa032a2b34f32da06" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40465 } } peers { permanent_uuid: "85ac5a425d1d44d793797f6f5e25dae5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42011 } } peers { permanent_uuid: "ddad36ae0ddf437f9f8c6f86a538b26a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34477 } }
I20260929 19:00:53.603158 1503 catalog_manager.cc:5495] T c95af5937e8046cfb96b47695df7cb15 P a4b5b67825d8469fa032a2b34f32da06 reported cstate change: term changed from 0 to 1, leader changed from <none> to a4b5b67825d8469fa032a2b34f32da06 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "a4b5b67825d8469fa032a2b34f32da06" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4b5b67825d8469fa032a2b34f32da06" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "85ac5a425d1d44d793797f6f5e25dae5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42011 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ddad36ae0ddf437f9f8c6f86a538b26a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34477 } health_report { overall_health: UNKNOWN } } }
I20260929 19:00:53.677569 2006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:53.675011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48202 (local address 127.0.98.3:40465)
0929 19:00:53.675094 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0929 19:00:53.675097 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 19:00:53.675108 (+ 11us) server_negotiation.cc:407] Connection header received
0929 19:00:53.675152 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:00:53.675153 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:00:53.675185 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:00:53.675240 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 19:00:53.675368 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:53.675883 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:53.676743 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:53.677020 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:53.677074 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:53.677240 (+ 166us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 19:00:53.677326 (+ 86us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 19:00:53.677400 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 19:00:53.677445 (+ 45us) server_negotiation.cc:299] Negotiation successful
0929 19:00:53.677463 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260929 19:00:53.679322 1801 raft_consensus.cc:1261] T c95af5937e8046cfb96b47695df7cb15 P 85ac5a425d1d44d793797f6f5e25dae5 [term 1 FOLLOWER]: Refusing update from remote peer a4b5b67825d8469fa032a2b34f32da06: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 19:00:53.679320 1661 raft_consensus.cc:1261] T c95af5937e8046cfb96b47695df7cb15 P ddad36ae0ddf437f9f8c6f86a538b26a [term 1 FOLLOWER]: Refusing update from remote peer a4b5b67825d8469fa032a2b34f32da06: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 19:00:53.679630 2016 consensus_queue.cc:1035] T c95af5937e8046cfb96b47695df7cb15 P a4b5b67825d8469fa032a2b34f32da06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85ac5a425d1d44d793797f6f5e25dae5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 19:00:53.679832 2020 consensus_queue.cc:1035] T c95af5937e8046cfb96b47695df7cb15 P a4b5b67825d8469fa032a2b34f32da06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddad36ae0ddf437f9f8c6f86a538b26a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34477 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 19:00:53.682474 2024 mvcc.cc:205] Tried to move back new op lower bound from 7334741826267893760 to 7334741825956249600. Current Snapshot: MvccSnapshot[applied={T|T < 7334741826267893760}]
I20260929 19:00:53.685274 1503 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44554:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 19:00:53.685376 1503 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44554:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 19:00:53.686056 1503 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4005b06255574d27844b7b7b093be143: Sending DeleteTablet for 3 replicas of tablet c95af5937e8046cfb96b47695df7cb15
I20260929 19:00:53.686324 1921 tablet_service.cc:1514] Processing DeleteTablet for tablet c95af5937e8046cfb96b47695df7cb15 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:00:53 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48198
I20260929 19:00:53.686429 1781 tablet_service.cc:1514] Processing DeleteTablet for tablet c95af5937e8046cfb96b47695df7cb15 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:00:53 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55752
I20260929 19:00:53.686435 1641 tablet_service.cc:1514] Processing DeleteTablet for tablet c95af5937e8046cfb96b47695df7cb15 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:00:53 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36098
I20260929 19:00:53.686618 2035 tablet_replica.cc:330] T c95af5937e8046cfb96b47695df7cb15 P ddad36ae0ddf437f9f8c6f86a538b26a: stopping tablet replica
I20260929 19:00:53.686713 2035 raft_consensus.cc:2229] T c95af5937e8046cfb96b47695df7cb15 P ddad36ae0ddf437f9f8c6f86a538b26a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 19:00:53.686774 2033 tablet_replica.cc:330] T c95af5937e8046cfb96b47695df7cb15 P a4b5b67825d8469fa032a2b34f32da06: stopping tablet replica
I20260929 19:00:53.686717 1503 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44554:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 19:00:53.686873 1503 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 19:00:53.686955 2033 raft_consensus.cc:2229] T c95af5937e8046cfb96b47695df7cb15 P a4b5b67825d8469fa032a2b34f32da06 [term 1 LEADER]: Raft consensus shutting down.
I20260929 19:00:53.687080 2033 raft_consensus.cc:2258] T c95af5937e8046cfb96b47695df7cb15 P a4b5b67825d8469fa032a2b34f32da06 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 19:00:53.687204 2035 raft_consensus.cc:2258] T c95af5937e8046cfb96b47695df7cb15 P ddad36ae0ddf437f9f8c6f86a538b26a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 19:00:53.687518 2033 ts_tablet_manager.cc:1905] T c95af5937e8046cfb96b47695df7cb15 P a4b5b67825d8469fa032a2b34f32da06: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 19:00:53.687574 2035 ts_tablet_manager.cc:1905] T c95af5937e8046cfb96b47695df7cb15 P ddad36ae0ddf437f9f8c6f86a538b26a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 19:00:53.688520 1641 tablet_service.cc:1467] Processing CreateTablet for tablet 2848f7f03a6d4f169f8f506b1a924014 (DEFAULT_TABLE table=test-table [id=77b84003350d49f68f29dac77eb9fd43]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:00:53.688642 1641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2848f7f03a6d4f169f8f506b1a924014. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:00:53.689235 2035 ts_tablet_manager.cc:1918] T c95af5937e8046cfb96b47695df7cb15 P ddad36ae0ddf437f9f8c6f86a538b26a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 19:00:53.689311 2035 log.cc:1196] T c95af5937e8046cfb96b47695df7cb15 P ddad36ae0ddf437f9f8c6f86a538b26a: Deleting WAL directory at /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708447806542-392-0/minicluster-data/ts-0/wal/wals/c95af5937e8046cfb96b47695df7cb15
I20260929 19:00:53.689525 2033 ts_tablet_manager.cc:1918] T c95af5937e8046cfb96b47695df7cb15 P a4b5b67825d8469fa032a2b34f32da06: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 19:00:53.689575 2035 ts_tablet_manager.cc:1939] T c95af5937e8046cfb96b47695df7cb15 P ddad36ae0ddf437f9f8c6f86a538b26a: Deleting consensus metadata
I20260929 19:00:53.689625 2033 log.cc:1196] T c95af5937e8046cfb96b47695df7cb15 P a4b5b67825d8469fa032a2b34f32da06: Deleting WAL directory at /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708447806542-392-0/minicluster-data/ts-2/wal/wals/c95af5937e8046cfb96b47695df7cb15
I20260929 19:00:53.689900 2033 ts_tablet_manager.cc:1939] T c95af5937e8046cfb96b47695df7cb15 P a4b5b67825d8469fa032a2b34f32da06: Deleting consensus metadata
I20260929 19:00:53.689918 2034 tablet_replica.cc:330] T c95af5937e8046cfb96b47695df7cb15 P 85ac5a425d1d44d793797f6f5e25dae5: stopping tablet replica
I20260929 19:00:53.689989 2013 tablet_bootstrap.cc:492] T 2848f7f03a6d4f169f8f506b1a924014 P ddad36ae0ddf437f9f8c6f86a538b26a: Bootstrap starting.
I20260929 19:00:53.690030 2034 raft_consensus.cc:2229] T c95af5937e8046cfb96b47695df7cb15 P 85ac5a425d1d44d793797f6f5e25dae5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 19:00:53.690126 2034 raft_consensus.cc:2258] T c95af5937e8046cfb96b47695df7cb15 P 85ac5a425d1d44d793797f6f5e25dae5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 19:00:53.690371 2013 tablet_bootstrap.cc:654] T 2848f7f03a6d4f169f8f506b1a924014 P ddad36ae0ddf437f9f8c6f86a538b26a: Neither blocks nor log segments found. Creating new log.
I20260929 19:00:53.690496 2034 ts_tablet_manager.cc:1905] T c95af5937e8046cfb96b47695df7cb15 P 85ac5a425d1d44d793797f6f5e25dae5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 19:00:53.690898 1488 catalog_manager.cc:4841] TS a4b5b67825d8469fa032a2b34f32da06 (127.0.98.3:40465): tablet c95af5937e8046cfb96b47695df7cb15 (table test-table [id=dc4d1874e5ab4873a3068cff7124f23d]) successfully deleted
I20260929 19:00:53.690973 2013 tablet_bootstrap.cc:492] T 2848f7f03a6d4f169f8f506b1a924014 P ddad36ae0ddf437f9f8c6f86a538b26a: No bootstrap required, opened a new log
I20260929 19:00:53.691012 2013 ts_tablet_manager.cc:1397] T 2848f7f03a6d4f169f8f506b1a924014 P ddad36ae0ddf437f9f8c6f86a538b26a: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260929 19:00:53.691104 1921 tablet_service.cc:1467] Processing CreateTablet for tablet 2848f7f03a6d4f169f8f506b1a924014 (DEFAULT_TABLE table=test-table [id=77b84003350d49f68f29dac77eb9fd43]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:00:53.691186 1921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2848f7f03a6d4f169f8f506b1a924014. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:00:53.691144 2013 raft_consensus.cc:348] T 2848f7f03a6d4f169f8f506b1a924014 P ddad36ae0ddf437f9f8c6f86a538b26a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddad36ae0ddf437f9f8c6f86a538b26a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34477 } } peers { permanent_uuid: "85ac5a425d1d44d793797f6f5e25dae5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42011 } } peers { permanent_uuid: "a4b5b67825d8469fa032a2b34f32da06" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40465 } }
I20260929 19:00:53.691210 2013 raft_consensus.cc:374] T 2848f7f03a6d4f169f8f506b1a924014 P ddad36ae0ddf437f9f8c6f86a538b26a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:00:53.691231 2013 raft_consensus.cc:729] T 2848f7f03a6d4f169f8f506b1a924014 P ddad36ae0ddf437f9f8c6f86a538b26a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddad36ae0ddf437f9f8c6f86a538b26a, State: Initialized, Role: FOLLOWER
I20260929 19:00:53.691277 2013 consensus_queue.cc:260] T 2848f7f03a6d4f169f8f506b1a924014 P ddad36ae0ddf437f9f8c6f86a538b26a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddad36ae0ddf437f9f8c6f86a538b26a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34477 } } peers { permanent_uuid: "85ac5a425d1d44d793797f6f5e25dae5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42011 } } peers { permanent_uuid: "a4b5b67825d8469fa032a2b34f32da06" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40465 } }
I20260929 19:00:53.691376 2013 ts_tablet_manager.cc:1428] T 2848f7f03a6d4f169f8f506b1a924014 P ddad36ae0ddf437f9f8c6f86a538b26a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260929 19:00:53.691975 2034 ts_tablet_manager.cc:1918] T c95af5937e8046cfb96b47695df7cb15 P 85ac5a425d1d44d793797f6f5e25dae5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 19:00:53.692046 2034 log.cc:1196] T c95af5937e8046cfb96b47695df7cb15 P 85ac5a425d1d44d793797f6f5e25dae5: Deleting WAL directory at /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708447806542-392-0/minicluster-data/ts-1/wal/wals/c95af5937e8046cfb96b47695df7cb15
I20260929 19:00:53.692286 2034 ts_tablet_manager.cc:1939] T c95af5937e8046cfb96b47695df7cb15 P 85ac5a425d1d44d793797f6f5e25dae5: Deleting consensus metadata
I20260929 19:00:53.692361 2010 tablet_bootstrap.cc:492] T 2848f7f03a6d4f169f8f506b1a924014 P a4b5b67825d8469fa032a2b34f32da06: Bootstrap starting.
I20260929 19:00:53.692440 1489 catalog_manager.cc:4841] TS ddad36ae0ddf437f9f8c6f86a538b26a (127.0.98.1:34477): tablet c95af5937e8046cfb96b47695df7cb15 (table test-table [id=dc4d1874e5ab4873a3068cff7124f23d]) successfully deleted
I20260929 19:00:53.692569 1781 tablet_service.cc:1467] Processing CreateTablet for tablet 2848f7f03a6d4f169f8f506b1a924014 (DEFAULT_TABLE table=test-table [id=77b84003350d49f68f29dac77eb9fd43]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:00:53.692684 1781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2848f7f03a6d4f169f8f506b1a924014. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:00:53.692718 2010 tablet_bootstrap.cc:654] T 2848f7f03a6d4f169f8f506b1a924014 P a4b5b67825d8469fa032a2b34f32da06: Neither blocks nor log segments found. Creating new log.
I20260929 19:00:53.693353 1489 catalog_manager.cc:4841] TS 85ac5a425d1d44d793797f6f5e25dae5 (127.0.98.2:42011): tablet c95af5937e8046cfb96b47695df7cb15 (table test-table [id=dc4d1874e5ab4873a3068cff7124f23d]) successfully deleted
I20260929 19:00:53.693760 2010 tablet_bootstrap.cc:492] T 2848f7f03a6d4f169f8f506b1a924014 P a4b5b67825d8469fa032a2b34f32da06: No bootstrap required, opened a new log
I20260929 19:00:53.693819 2010 ts_tablet_manager.cc:1397] T 2848f7f03a6d4f169f8f506b1a924014 P a4b5b67825d8469fa032a2b34f32da06: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260929 19:00:53.693867 2011 tablet_bootstrap.cc:492] T 2848f7f03a6d4f169f8f506b1a924014 P 85ac5a425d1d44d793797f6f5e25dae5: Bootstrap starting.
I20260929 19:00:53.693954 2010 raft_consensus.cc:348] T 2848f7f03a6d4f169f8f506b1a924014 P a4b5b67825d8469fa032a2b34f32da06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddad36ae0ddf437f9f8c6f86a538b26a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34477 } } peers { permanent_uuid: "85ac5a425d1d44d793797f6f5e25dae5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42011 } } peers { permanent_uuid: "a4b5b67825d8469fa032a2b34f32da06" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40465 } }
I20260929 19:00:53.694037 2010 raft_consensus.cc:374] T 2848f7f03a6d4f169f8f506b1a924014 P a4b5b67825d8469fa032a2b34f32da06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:00:53.694067 2010 raft_consensus.cc:729] T 2848f7f03a6d4f169f8f506b1a924014 P a4b5b67825d8469fa032a2b34f32da06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4b5b67825d8469fa032a2b34f32da06, State: Initialized, Role: FOLLOWER
I20260929 19:00:53.694111 2010 consensus_queue.cc:260] T 2848f7f03a6d4f169f8f506b1a924014 P a4b5b67825d8469fa032a2b34f32da06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddad36ae0ddf437f9f8c6f86a538b26a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34477 } } peers { permanent_uuid: "85ac5a425d1d44d793797f6f5e25dae5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42011 } } peers { permanent_uuid: "a4b5b67825d8469fa032a2b34f32da06" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40465 } }
I20260929 19:00:53.694216 2010 ts_tablet_manager.cc:1428] T 2848f7f03a6d4f169f8f506b1a924014 P a4b5b67825d8469fa032a2b34f32da06: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260929 19:00:53.694231 2011 tablet_bootstrap.cc:654] T 2848f7f03a6d4f169f8f506b1a924014 P 85ac5a425d1d44d793797f6f5e25dae5: Neither blocks nor log segments found. Creating new log.
I20260929 19:00:53.694794 2011 tablet_bootstrap.cc:492] T 2848f7f03a6d4f169f8f506b1a924014 P 85ac5a425d1d44d793797f6f5e25dae5: No bootstrap required, opened a new log
I20260929 19:00:53.694847 2011 ts_tablet_manager.cc:1397] T 2848f7f03a6d4f169f8f506b1a924014 P 85ac5a425d1d44d793797f6f5e25dae5: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260929 19:00:53.694970 2011 raft_consensus.cc:348] T 2848f7f03a6d4f169f8f506b1a924014 P 85ac5a425d1d44d793797f6f5e25dae5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddad36ae0ddf437f9f8c6f86a538b26a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34477 } } peers { permanent_uuid: "85ac5a425d1d44d793797f6f5e25dae5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42011 } } peers { permanent_uuid: "a4b5b67825d8469fa032a2b34f32da06" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40465 } }
I20260929 19:00:53.695020 2011 raft_consensus.cc:374] T 2848f7f03a6d4f169f8f506b1a924014 P 85ac5a425d1d44d793797f6f5e25dae5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:00:53.695034 2011 raft_consensus.cc:729] T 2848f7f03a6d4f169f8f506b1a924014 P 85ac5a425d1d44d793797f6f5e25dae5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85ac5a425d1d44d793797f6f5e25dae5, State: Initialized, Role: FOLLOWER
I20260929 19:00:53.695087 2011 consensus_queue.cc:260] T 2848f7f03a6d4f169f8f506b1a924014 P 85ac5a425d1d44d793797f6f5e25dae5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddad36ae0ddf437f9f8c6f86a538b26a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34477 } } peers { permanent_uuid: "85ac5a425d1d44d793797f6f5e25dae5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42011 } } peers { permanent_uuid: "a4b5b67825d8469fa032a2b34f32da06" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40465 } }
I20260929 19:00:53.695353 2011 ts_tablet_manager.cc:1428] T 2848f7f03a6d4f169f8f506b1a924014 P 85ac5a425d1d44d793797f6f5e25dae5: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260929 19:00:53.976622 2017 raft_consensus.cc:482] T 2848f7f03a6d4f169f8f506b1a924014 P ddad36ae0ddf437f9f8c6f86a538b26a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 19:00:53.976699 2017 raft_consensus.cc:504] T 2848f7f03a6d4f169f8f506b1a924014 P ddad36ae0ddf437f9f8c6f86a538b26a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddad36ae0ddf437f9f8c6f86a538b26a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34477 } } peers { permanent_uuid: "85ac5a425d1d44d793797f6f5e25dae5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42011 } } peers { permanent_uuid: "a4b5b67825d8469fa032a2b34f32da06" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40465 } }
I20260929 19:00:53.977121 2017 leader_election.cc:288] T 2848f7f03a6d4f169f8f506b1a924014 P ddad36ae0ddf437f9f8c6f86a538b26a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85ac5a425d1d44d793797f6f5e25dae5 (127.0.98.2:42011), a4b5b67825d8469fa032a2b34f32da06 (127.0.98.3:40465)
I20260929 19:00:53.980115 2006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:53.977333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39073 (local address 127.0.98.3:40465)
0929 19:00:53.977379 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0929 19:00:53.977381 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:00:53.977514 (+ 133us) server_negotiation.cc:407] Connection header received
0929 19:00:53.977612 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:00:53.977613 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:00:53.977622 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:00:53.977649 (+ 27us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:00:53.977829 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:53.978551 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:53.979423 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:53.979800 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:53.979881 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:53.979892 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0929 19:00:53.980023 (+ 131us) server_negotiation.cc:299] Negotiation successful
0929 19:00:53.980055 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260929 19:00:53.980135 2039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:53.977246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40465 (local address 127.0.98.1:39073)
0929 19:00:53.977457 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0929 19:00:53.977466 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 19:00:53.977537 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:00:53.977705 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:00:53.977706 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:00:53.977707 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:00:53.977759 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:00:53.977761 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:00:53.978618 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:00:53.978619 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:00:53.979353 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:00:53.979354 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:00:53.979881 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:00:53.979882 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:00:53.979943 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:53.979947 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 19:00:53.980016 (+ 69us) client_negotiation.cc:240] Negotiation successful
0929 19:00:53.980052 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"mutex_wait_us":8,"thread_start_us":82,"threads_started":1}
I20260929 19:00:53.980400 1941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2848f7f03a6d4f169f8f506b1a924014" candidate_uuid: "ddad36ae0ddf437f9f8c6f86a538b26a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4b5b67825d8469fa032a2b34f32da06" is_pre_election: true
I20260929 19:00:53.980520 1941 raft_consensus.cc:2454] T 2848f7f03a6d4f169f8f506b1a924014 P a4b5b67825d8469fa032a2b34f32da06 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ddad36ae0ddf437f9f8c6f86a538b26a in term 0.
I20260929 19:00:53.980702 1592 leader_election.cc:302] T 2848f7f03a6d4f169f8f506b1a924014 P ddad36ae0ddf437f9f8c6f86a538b26a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a4b5b67825d8469fa032a2b34f32da06, ddad36ae0ddf437f9f8c6f86a538b26a; no voters:
I20260929 19:00:53.980836 2017 raft_consensus.cc:2781] T 2848f7f03a6d4f169f8f506b1a924014 P ddad36ae0ddf437f9f8c6f86a538b26a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 19:00:53.980891 2017 raft_consensus.cc:482] T 2848f7f03a6d4f169f8f506b1a924014 P ddad36ae0ddf437f9f8c6f86a538b26a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 19:00:53.980952 2017 raft_consensus.cc:3037] T 2848f7f03a6d4f169f8f506b1a924014 P ddad36ae0ddf437f9f8c6f86a538b26a [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:00:53.981170 2040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:53.977245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42011 (local address 127.0.98.1:53867)
0929 19:00:53.977418 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0929 19:00:53.977428 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 19:00:53.977512 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:00:53.977636 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:00:53.977637 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:00:53.977639 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:00:53.977692 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:00:53.977693 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:00:53.979497 (+ 1804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:00:53.979497 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:00:53.980189 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:00:53.980192 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:00:53.980987 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:00:53.980988 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:00:53.981050 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:53.981052 (+ 2us) client_negotiation.cc:769] Sending connection context
0929 19:00:53.981091 (+ 39us) client_negotiation.cc:240] Negotiation successful
0929 19:00:53.981117 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":49,"threads_started":1}
I20260929 19:00:53.981226 2008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:53.977394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53867 (local address 127.0.98.2:42011)
0929 19:00:53.977446 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0929 19:00:53.977447 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 19:00:53.977488 (+ 41us) server_negotiation.cc:407] Connection header received
0929 19:00:53.977573 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:00:53.977574 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:00:53.977584 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:00:53.977631 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:00:53.977763 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:53.978545 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:53.980333 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:53.980851 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:53.980919 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:53.980929 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0929 19:00:53.981123 (+ 194us) server_negotiation.cc:299] Negotiation successful
0929 19:00:53.981168 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260929 19:00:53.981405 1801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2848f7f03a6d4f169f8f506b1a924014" candidate_uuid: "ddad36ae0ddf437f9f8c6f86a538b26a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85ac5a425d1d44d793797f6f5e25dae5" is_pre_election: true
I20260929 19:00:53.981521 1801 raft_consensus.cc:2454] T 2848f7f03a6d4f169f8f506b1a924014 P 85ac5a425d1d44d793797f6f5e25dae5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ddad36ae0ddf437f9f8c6f86a538b26a in term 0.
I20260929 19:00:53.981817 2017 raft_consensus.cc:504] T 2848f7f03a6d4f169f8f506b1a924014 P ddad36ae0ddf437f9f8c6f86a538b26a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddad36ae0ddf437f9f8c6f86a538b26a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34477 } } peers { permanent_uuid: "85ac5a425d1d44d793797f6f5e25dae5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42011 } } peers { permanent_uuid: "a4b5b67825d8469fa032a2b34f32da06" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40465 } }
I20260929 19:00:53.981945 2017 leader_election.cc:288] T 2848f7f03a6d4f169f8f506b1a924014 P ddad36ae0ddf437f9f8c6f86a538b26a [CANDIDATE]: Term 1 election: Requested vote from peers 85ac5a425d1d44d793797f6f5e25dae5 (127.0.98.2:42011), a4b5b67825d8469fa032a2b34f32da06 (127.0.98.3:40465)
I20260929 19:00:53.982072 1801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2848f7f03a6d4f169f8f506b1a924014" candidate_uuid: "ddad36ae0ddf437f9f8c6f86a538b26a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85ac5a425d1d44d793797f6f5e25dae5"
I20260929 19:00:53.982092 1941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2848f7f03a6d4f169f8f506b1a924014" candidate_uuid: "ddad36ae0ddf437f9f8c6f86a538b26a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4b5b67825d8469fa032a2b34f32da06"
I20260929 19:00:53.982151 1801 raft_consensus.cc:3037] T 2848f7f03a6d4f169f8f506b1a924014 P 85ac5a425d1d44d793797f6f5e25dae5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:00:53.982151 1941 raft_consensus.cc:3037] T 2848f7f03a6d4f169f8f506b1a924014 P a4b5b67825d8469fa032a2b34f32da06 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:00:53.982868 1801 raft_consensus.cc:2454] T 2848f7f03a6d4f169f8f506b1a924014 P 85ac5a425d1d44d793797f6f5e25dae5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ddad36ae0ddf437f9f8c6f86a538b26a in term 1.
I20260929 19:00:53.982939 1941 raft_consensus.cc:2454] T 2848f7f03a6d4f169f8f506b1a924014 P a4b5b67825d8469fa032a2b34f32da06 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ddad36ae0ddf437f9f8c6f86a538b26a in term 1.
I20260929 19:00:53.983060 1593 leader_election.cc:302] T 2848f7f03a6d4f169f8f506b1a924014 P ddad36ae0ddf437f9f8c6f86a538b26a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85ac5a425d1d44d793797f6f5e25dae5, ddad36ae0ddf437f9f8c6f86a538b26a; no voters:
I20260929 19:00:53.983175 2017 raft_consensus.cc:2781] T 2848f7f03a6d4f169f8f506b1a924014 P ddad36ae0ddf437f9f8c6f86a538b26a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:00:53.983314 2017 raft_consensus.cc:686] T 2848f7f03a6d4f169f8f506b1a924014 P ddad36ae0ddf437f9f8c6f86a538b26a [term 1 LEADER]: Becoming Leader. State: Replica: ddad36ae0ddf437f9f8c6f86a538b26a, State: Running, Role: LEADER
I20260929 19:00:53.983397 2017 consensus_queue.cc:237] T 2848f7f03a6d4f169f8f506b1a924014 P ddad36ae0ddf437f9f8c6f86a538b26a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddad36ae0ddf437f9f8c6f86a538b26a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34477 } } peers { permanent_uuid: "85ac5a425d1d44d793797f6f5e25dae5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42011 } } peers { permanent_uuid: "a4b5b67825d8469fa032a2b34f32da06" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40465 } }
I20260929 19:00:53.984030 1503 catalog_manager.cc:5495] T 2848f7f03a6d4f169f8f506b1a924014 P ddad36ae0ddf437f9f8c6f86a538b26a reported cstate change: term changed from 0 to 1, leader changed from <none> to ddad36ae0ddf437f9f8c6f86a538b26a (127.0.98.1). New cstate: current_term: 1 leader_uuid: "ddad36ae0ddf437f9f8c6f86a538b26a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddad36ae0ddf437f9f8c6f86a538b26a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34477 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "85ac5a425d1d44d793797f6f5e25dae5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42011 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4b5b67825d8469fa032a2b34f32da06" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40465 } health_report { overall_health: UNKNOWN } } }
Sep 29 19:00:54 dist-test-slave-lqkv krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708452, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 19:00:54.072768 2048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:54.061375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39367 (local address 127.0.0.1:44562)
0929 19:00:54.061542 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0929 19:00:54.061553 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 19:00:54.061618 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:00:54.061746 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:00:54.061746 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:00:54.062032 (+ 286us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:00:54.062072 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:00:54.062074 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:00:54.062645 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:00:54.062645 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:00:54.063137 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:00:54.063138 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:00:54.063422 (+ 284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:00:54.063423 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:00:54.063458 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:54.069084 (+ 5626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:00:54.069088 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:00:54.070404 (+ 1316us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:00:54.071964 (+ 1560us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:00:54.071969 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:00:54.071971 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:00:54.072134 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:00:54.072292 (+ 158us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:00:54.072292 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:00:54.072293 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:00:54.072339 (+ 46us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:00:54.072524 (+ 185us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:00:54.072529 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:00:54.072563 (+ 34us) client_negotiation.cc:769] Sending connection context
0929 19:00:54.072609 (+ 46us) client_negotiation.cc:240] Negotiation successful
0929 19:00:54.072686 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":83,"threads_started":1}
I20260929 19:00:54.072883 2049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:54.061429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44562 (local address 127.0.98.62:39367)
0929 19:00:54.061576 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0929 19:00:54.061578 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:00:54.061607 (+ 29us) server_negotiation.cc:407] Connection header received
0929 19:00:54.061667 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:00:54.061667 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:00:54.061676 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:00:54.061733 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:00:54.062134 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:54.062589 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:54.063200 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:54.063369 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:54.063413 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:54.070481 (+ 7068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:00:54.070482 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:00:54.070483 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:00:54.070488 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:00:54.071896 (+ 1408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:00:54.072192 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:00:54.072193 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:00:54.072193 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:00:54.072242 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:00:54.072387 (+ 145us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:00:54.072388 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:00:54.072388 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:00:54.072478 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:00:54.072525 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0929 19:00:54.072626 (+ 101us) server_negotiation.cc:299] Negotiation successful
0929 19:00:54.072729 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":41,"server-negotiator.queue_time_us":125,"thread_start_us":93,"threads_started":1}
I20260929 19:00:54.074795 1503 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44562:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260929 19:00:54.074990 1503 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 19:00:54.076365 1641 tablet_service.cc:1467] Processing CreateTablet for tablet b7f270fb6efd4a56b0fd4f362e925111 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7eb65cee4b754760bae848bd9008a18e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 19:00:54.076349 1921 tablet_service.cc:1467] Processing CreateTablet for tablet b7f270fb6efd4a56b0fd4f362e925111 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7eb65cee4b754760bae848bd9008a18e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 19:00:54.076395 1781 tablet_service.cc:1467] Processing CreateTablet for tablet b7f270fb6efd4a56b0fd4f362e925111 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7eb65cee4b754760bae848bd9008a18e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 19:00:54.076517 1781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7f270fb6efd4a56b0fd4f362e925111. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:00:54.076519 1921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7f270fb6efd4a56b0fd4f362e925111. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:00:54.076525 1641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7f270fb6efd4a56b0fd4f362e925111. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:00:54.077911 2013 tablet_bootstrap.cc:492] T b7f270fb6efd4a56b0fd4f362e925111 P ddad36ae0ddf437f9f8c6f86a538b26a: Bootstrap starting.
I20260929 19:00:54.078039 2010 tablet_bootstrap.cc:492] T b7f270fb6efd4a56b0fd4f362e925111 P a4b5b67825d8469fa032a2b34f32da06: Bootstrap starting.
I20260929 19:00:54.078047 2011 tablet_bootstrap.cc:492] T b7f270fb6efd4a56b0fd4f362e925111 P 85ac5a425d1d44d793797f6f5e25dae5: Bootstrap starting.
I20260929 19:00:54.078320 2013 tablet_bootstrap.cc:654] T b7f270fb6efd4a56b0fd4f362e925111 P ddad36ae0ddf437f9f8c6f86a538b26a: Neither blocks nor log segments found. Creating new log.
I20260929 19:00:54.078799 2010 tablet_bootstrap.cc:654] T b7f270fb6efd4a56b0fd4f362e925111 P a4b5b67825d8469fa032a2b34f32da06: Neither blocks nor log segments found. Creating new log.
I20260929 19:00:54.078887 2011 tablet_bootstrap.cc:654] T b7f270fb6efd4a56b0fd4f362e925111 P 85ac5a425d1d44d793797f6f5e25dae5: Neither blocks nor log segments found. Creating new log.
I20260929 19:00:54.079341 2010 tablet_bootstrap.cc:492] T b7f270fb6efd4a56b0fd4f362e925111 P a4b5b67825d8469fa032a2b34f32da06: No bootstrap required, opened a new log
I20260929 19:00:54.079406 2010 ts_tablet_manager.cc:1397] T b7f270fb6efd4a56b0fd4f362e925111 P a4b5b67825d8469fa032a2b34f32da06: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260929 19:00:54.079470 2011 tablet_bootstrap.cc:492] T b7f270fb6efd4a56b0fd4f362e925111 P 85ac5a425d1d44d793797f6f5e25dae5: No bootstrap required, opened a new log
I20260929 19:00:54.079516 2011 ts_tablet_manager.cc:1397] T b7f270fb6efd4a56b0fd4f362e925111 P 85ac5a425d1d44d793797f6f5e25dae5: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.001s
I20260929 19:00:54.079527 2010 raft_consensus.cc:348] T b7f270fb6efd4a56b0fd4f362e925111 P a4b5b67825d8469fa032a2b34f32da06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddad36ae0ddf437f9f8c6f86a538b26a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34477 } } peers { permanent_uuid: "85ac5a425d1d44d793797f6f5e25dae5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42011 } } peers { permanent_uuid: "a4b5b67825d8469fa032a2b34f32da06" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40465 } }
I20260929 19:00:54.079609 2010 raft_consensus.cc:374] T b7f270fb6efd4a56b0fd4f362e925111 P a4b5b67825d8469fa032a2b34f32da06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:00:54.079636 2010 raft_consensus.cc:729] T b7f270fb6efd4a56b0fd4f362e925111 P a4b5b67825d8469fa032a2b34f32da06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4b5b67825d8469fa032a2b34f32da06, State: Initialized, Role: FOLLOWER
I20260929 19:00:54.079641 2011 raft_consensus.cc:348] T b7f270fb6efd4a56b0fd4f362e925111 P 85ac5a425d1d44d793797f6f5e25dae5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddad36ae0ddf437f9f8c6f86a538b26a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34477 } } peers { permanent_uuid: "85ac5a425d1d44d793797f6f5e25dae5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42011 } } peers { permanent_uuid: "a4b5b67825d8469fa032a2b34f32da06" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40465 } }
I20260929 19:00:54.079715 2011 raft_consensus.cc:374] T b7f270fb6efd4a56b0fd4f362e925111 P 85ac5a425d1d44d793797f6f5e25dae5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:00:54.079689 2010 consensus_queue.cc:260] T b7f270fb6efd4a56b0fd4f362e925111 P a4b5b67825d8469fa032a2b34f32da06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddad36ae0ddf437f9f8c6f86a538b26a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34477 } } peers { permanent_uuid: "85ac5a425d1d44d793797f6f5e25dae5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42011 } } peers { permanent_uuid: "a4b5b67825d8469fa032a2b34f32da06" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40465 } }
I20260929 19:00:54.079737 2011 raft_consensus.cc:729] T b7f270fb6efd4a56b0fd4f362e925111 P 85ac5a425d1d44d793797f6f5e25dae5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85ac5a425d1d44d793797f6f5e25dae5, State: Initialized, Role: FOLLOWER
I20260929 19:00:54.079800 2010 ts_tablet_manager.cc:1428] T b7f270fb6efd4a56b0fd4f362e925111 P a4b5b67825d8469fa032a2b34f32da06: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260929 19:00:54.079778 2011 consensus_queue.cc:260] T b7f270fb6efd4a56b0fd4f362e925111 P 85ac5a425d1d44d793797f6f5e25dae5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddad36ae0ddf437f9f8c6f86a538b26a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34477 } } peers { permanent_uuid: "85ac5a425d1d44d793797f6f5e25dae5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42011 } } peers { permanent_uuid: "a4b5b67825d8469fa032a2b34f32da06" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40465 } }
I20260929 19:00:54.079874 2011 ts_tablet_manager.cc:1428] T b7f270fb6efd4a56b0fd4f362e925111 P 85ac5a425d1d44d793797f6f5e25dae5: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260929 19:00:54.079833 2028 tablet_replica.cc:439] T b7f270fb6efd4a56b0fd4f362e925111 P a4b5b67825d8469fa032a2b34f32da06: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddad36ae0ddf437f9f8c6f86a538b26a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34477 } } peers { permanent_uuid: "85ac5a425d1d44d793797f6f5e25dae5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42011 } } peers { permanent_uuid: "a4b5b67825d8469fa032a2b34f32da06" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40465 } } }
I20260929 19:00:54.079922 2028 tablet_replica.cc:442] T b7f270fb6efd4a56b0fd4f362e925111 P a4b5b67825d8469fa032a2b34f32da06: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 19:00:54.080021 2018 tablet_replica.cc:439] T b7f270fb6efd4a56b0fd4f362e925111 P 85ac5a425d1d44d793797f6f5e25dae5: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddad36ae0ddf437f9f8c6f86a538b26a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34477 } } peers { permanent_uuid: "85ac5a425d1d44d793797f6f5e25dae5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42011 } } peers { permanent_uuid: "a4b5b67825d8469fa032a2b34f32da06" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40465 } } }
I20260929 19:00:54.080111 2018 tablet_replica.cc:442] T b7f270fb6efd4a56b0fd4f362e925111 P 85ac5a425d1d44d793797f6f5e25dae5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 19:00:54.080271 2013 tablet_bootstrap.cc:492] T b7f270fb6efd4a56b0fd4f362e925111 P ddad36ae0ddf437f9f8c6f86a538b26a: No bootstrap required, opened a new log
I20260929 19:00:54.080336 2013 ts_tablet_manager.cc:1397] T b7f270fb6efd4a56b0fd4f362e925111 P ddad36ae0ddf437f9f8c6f86a538b26a: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260929 19:00:54.080484 2013 raft_consensus.cc:348] T b7f270fb6efd4a56b0fd4f362e925111 P ddad36ae0ddf437f9f8c6f86a538b26a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddad36ae0ddf437f9f8c6f86a538b26a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34477 } } peers { permanent_uuid: "85ac5a425d1d44d793797f6f5e25dae5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42011 } } peers { permanent_uuid: "a4b5b67825d8469fa032a2b34f32da06" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40465 } }
I20260929 19:00:54.080554 2013 raft_consensus.cc:374] T b7f270fb6efd4a56b0fd4f362e925111 P ddad36ae0ddf437f9f8c6f86a538b26a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:00:54.080583 2013 raft_consensus.cc:729] T b7f270fb6efd4a56b0fd4f362e925111 P ddad36ae0ddf437f9f8c6f86a538b26a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddad36ae0ddf437f9f8c6f86a538b26a, State: Initialized, Role: FOLLOWER
I20260929 19:00:54.080637 2013 consensus_queue.cc:260] T b7f270fb6efd4a56b0fd4f362e925111 P ddad36ae0ddf437f9f8c6f86a538b26a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddad36ae0ddf437f9f8c6f86a538b26a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34477 } } peers { permanent_uuid: "85ac5a425d1d44d793797f6f5e25dae5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42011 } } peers { permanent_uuid: "a4b5b67825d8469fa032a2b34f32da06" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40465 } }
I20260929 19:00:54.080760 2013 ts_tablet_manager.cc:1428] T b7f270fb6efd4a56b0fd4f362e925111 P ddad36ae0ddf437f9f8c6f86a538b26a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260929 19:00:54.080792 2017 tablet_replica.cc:439] T b7f270fb6efd4a56b0fd4f362e925111 P ddad36ae0ddf437f9f8c6f86a538b26a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddad36ae0ddf437f9f8c6f86a538b26a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34477 } } peers { permanent_uuid: "85ac5a425d1d44d793797f6f5e25dae5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42011 } } peers { permanent_uuid: "a4b5b67825d8469fa032a2b34f32da06" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40465 } } }
I20260929 19:00:54.080878 2017 tablet_replica.cc:442] T b7f270fb6efd4a56b0fd4f362e925111 P ddad36ae0ddf437f9f8c6f86a538b26a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 19:00:54.323732 2018 raft_consensus.cc:482] T b7f270fb6efd4a56b0fd4f362e925111 P 85ac5a425d1d44d793797f6f5e25dae5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 19:00:54.323834 2018 raft_consensus.cc:504] T b7f270fb6efd4a56b0fd4f362e925111 P 85ac5a425d1d44d793797f6f5e25dae5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddad36ae0ddf437f9f8c6f86a538b26a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34477 } } peers { permanent_uuid: "85ac5a425d1d44d793797f6f5e25dae5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42011 } } peers { permanent_uuid: "a4b5b67825d8469fa032a2b34f32da06" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40465 } }
I20260929 19:00:54.324208 2018 leader_election.cc:288] T b7f270fb6efd4a56b0fd4f362e925111 P 85ac5a425d1d44d793797f6f5e25dae5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ddad36ae0ddf437f9f8c6f86a538b26a (127.0.98.1:34477), a4b5b67825d8469fa032a2b34f32da06 (127.0.98.3:40465)
I20260929 19:00:54.327222 2006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:54.324487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48507 (local address 127.0.98.3:40465)
0929 19:00:54.324532 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0929 19:00:54.324534 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:00:54.324656 (+ 122us) server_negotiation.cc:407] Connection header received
0929 19:00:54.324740 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:00:54.324741 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:00:54.324752 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:00:54.324849 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:00:54.325002 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:54.325513 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:54.326522 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:54.326897 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:54.326949 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:54.326981 (+ 32us) server_negotiation.cc:1035] Waiting for connection context
0929 19:00:54.327119 (+ 138us) server_negotiation.cc:299] Negotiation successful
0929 19:00:54.327158 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260929 19:00:54.327221 2054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:54.324434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40465 (local address 127.0.98.2:48507)
0929 19:00:54.324568 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0929 19:00:54.324578 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 19:00:54.324649 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:00:54.324856 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:00:54.324857 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:00:54.324859 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:00:54.324938 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:00:54.324941 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:00:54.325591 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:00:54.325592 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:00:54.326433 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:00:54.326435 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:00:54.326975 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:00:54.326977 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:00:54.327042 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:54.327053 (+ 11us) client_negotiation.cc:769] Sending connection context
0929 19:00:54.327094 (+ 41us) client_negotiation.cc:240] Negotiation successful
0929 19:00:54.327123 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"mutex_wait_us":31,"thread_start_us":60,"threads_started":1}
I20260929 19:00:54.327301 2055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:54.324470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40497 (local address 127.0.98.1:34477)
0929 19:00:54.324565 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0929 19:00:54.324567 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:00:54.324642 (+ 75us) server_negotiation.cc:407] Connection header received
0929 19:00:54.324741 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:00:54.324741 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:00:54.324751 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:00:54.324850 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:00:54.325056 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:54.325518 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:54.326529 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:54.326905 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:54.326985 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:54.326995 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0929 19:00:54.327184 (+ 189us) server_negotiation.cc:299] Negotiation successful
0929 19:00:54.327224 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76,"thread_start_us":45,"threads_started":1}
I20260929 19:00:54.327328 2053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:54.324374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34477 (local address 127.0.98.2:40497)
0929 19:00:54.324554 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0929 19:00:54.324571 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 19:00:54.324653 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:00:54.324859 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:00:54.324860 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:00:54.324862 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:00:54.324979 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:00:54.324981 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:00:54.325619 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:00:54.325620 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:00:54.326436 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:00:54.326438 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:00:54.327008 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:00:54.327009 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:00:54.327100 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:54.327105 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 19:00:54.327161 (+ 56us) client_negotiation.cc:240] Negotiation successful
0929 19:00:54.327200 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"mutex_wait_us":61,"thread_start_us":73,"threads_started":1}
I20260929 19:00:54.327705 1941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7f270fb6efd4a56b0fd4f362e925111" candidate_uuid: "85ac5a425d1d44d793797f6f5e25dae5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4b5b67825d8469fa032a2b34f32da06" is_pre_election: true
I20260929 19:00:54.327715 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7f270fb6efd4a56b0fd4f362e925111" candidate_uuid: "85ac5a425d1d44d793797f6f5e25dae5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddad36ae0ddf437f9f8c6f86a538b26a" is_pre_election: true
I20260929 19:00:54.327832 1941 raft_consensus.cc:2454] T b7f270fb6efd4a56b0fd4f362e925111 P a4b5b67825d8469fa032a2b34f32da06 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85ac5a425d1d44d793797f6f5e25dae5 in term 0.
I20260929 19:00:54.327834 1661 raft_consensus.cc:2454] T b7f270fb6efd4a56b0fd4f362e925111 P ddad36ae0ddf437f9f8c6f86a538b26a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85ac5a425d1d44d793797f6f5e25dae5 in term 0.
I20260929 19:00:54.328015 1733 leader_election.cc:302] T b7f270fb6efd4a56b0fd4f362e925111 P 85ac5a425d1d44d793797f6f5e25dae5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85ac5a425d1d44d793797f6f5e25dae5, ddad36ae0ddf437f9f8c6f86a538b26a; no voters:
I20260929 19:00:54.328158 2018 raft_consensus.cc:2781] T b7f270fb6efd4a56b0fd4f362e925111 P 85ac5a425d1d44d793797f6f5e25dae5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 19:00:54.328197 2018 raft_consensus.cc:482] T b7f270fb6efd4a56b0fd4f362e925111 P 85ac5a425d1d44d793797f6f5e25dae5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 19:00:54.328217 2018 raft_consensus.cc:3037] T b7f270fb6efd4a56b0fd4f362e925111 P 85ac5a425d1d44d793797f6f5e25dae5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:00:54.328960 2018 raft_consensus.cc:504] T b7f270fb6efd4a56b0fd4f362e925111 P 85ac5a425d1d44d793797f6f5e25dae5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddad36ae0ddf437f9f8c6f86a538b26a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34477 } } peers { permanent_uuid: "85ac5a425d1d44d793797f6f5e25dae5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42011 } } peers { permanent_uuid: "a4b5b67825d8469fa032a2b34f32da06" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40465 } }
I20260929 19:00:54.329089 2018 leader_election.cc:288] T b7f270fb6efd4a56b0fd4f362e925111 P 85ac5a425d1d44d793797f6f5e25dae5 [CANDIDATE]: Term 1 election: Requested vote from peers ddad36ae0ddf437f9f8c6f86a538b26a (127.0.98.1:34477), a4b5b67825d8469fa032a2b34f32da06 (127.0.98.3:40465)
I20260929 19:00:54.329211 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7f270fb6efd4a56b0fd4f362e925111" candidate_uuid: "85ac5a425d1d44d793797f6f5e25dae5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddad36ae0ddf437f9f8c6f86a538b26a"
I20260929 19:00:54.329217 1941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7f270fb6efd4a56b0fd4f362e925111" candidate_uuid: "85ac5a425d1d44d793797f6f5e25dae5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4b5b67825d8469fa032a2b34f32da06"
I20260929 19:00:54.329275 1661 raft_consensus.cc:3037] T b7f270fb6efd4a56b0fd4f362e925111 P ddad36ae0ddf437f9f8c6f86a538b26a [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:00:54.329303 1941 raft_consensus.cc:3037] T b7f270fb6efd4a56b0fd4f362e925111 P a4b5b67825d8469fa032a2b34f32da06 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:00:54.329828 1661 raft_consensus.cc:2454] T b7f270fb6efd4a56b0fd4f362e925111 P ddad36ae0ddf437f9f8c6f86a538b26a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85ac5a425d1d44d793797f6f5e25dae5 in term 1.
I20260929 19:00:54.329981 1941 raft_consensus.cc:2454] T b7f270fb6efd4a56b0fd4f362e925111 P a4b5b67825d8469fa032a2b34f32da06 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85ac5a425d1d44d793797f6f5e25dae5 in term 1.
I20260929 19:00:54.329986 1733 leader_election.cc:302] T b7f270fb6efd4a56b0fd4f362e925111 P 85ac5a425d1d44d793797f6f5e25dae5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85ac5a425d1d44d793797f6f5e25dae5, ddad36ae0ddf437f9f8c6f86a538b26a; no voters:
I20260929 19:00:54.330063 2018 raft_consensus.cc:2781] T b7f270fb6efd4a56b0fd4f362e925111 P 85ac5a425d1d44d793797f6f5e25dae5 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:00:54.330147 2018 raft_consensus.cc:686] T b7f270fb6efd4a56b0fd4f362e925111 P 85ac5a425d1d44d793797f6f5e25dae5 [term 1 LEADER]: Becoming Leader. State: Replica: 85ac5a425d1d44d793797f6f5e25dae5, State: Running, Role: LEADER
I20260929 19:00:54.330215 2018 consensus_queue.cc:237] T b7f270fb6efd4a56b0fd4f362e925111 P 85ac5a425d1d44d793797f6f5e25dae5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddad36ae0ddf437f9f8c6f86a538b26a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34477 } } peers { permanent_uuid: "85ac5a425d1d44d793797f6f5e25dae5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42011 } } peers { permanent_uuid: "a4b5b67825d8469fa032a2b34f32da06" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40465 } }
I20260929 19:00:54.330523 2056 tablet_replica.cc:439] T b7f270fb6efd4a56b0fd4f362e925111 P 85ac5a425d1d44d793797f6f5e25dae5: TxnStatusTablet state changed. Reason: New leader 85ac5a425d1d44d793797f6f5e25dae5. Latest consensus state: current_term: 1 leader_uuid: "85ac5a425d1d44d793797f6f5e25dae5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddad36ae0ddf437f9f8c6f86a538b26a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34477 } } peers { permanent_uuid: "85ac5a425d1d44d793797f6f5e25dae5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42011 } } peers { permanent_uuid: "a4b5b67825d8469fa032a2b34f32da06" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40465 } } }
I20260929 19:00:54.330646 2056 tablet_replica.cc:442] T b7f270fb6efd4a56b0fd4f362e925111 P 85ac5a425d1d44d793797f6f5e25dae5: This TxnStatusTablet replica's current role is: LEADER
I20260929 19:00:54.330822 2058 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260929 19:00:54.330850 1504 catalog_manager.cc:5495] T b7f270fb6efd4a56b0fd4f362e925111 P 85ac5a425d1d44d793797f6f5e25dae5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 85ac5a425d1d44d793797f6f5e25dae5 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "85ac5a425d1d44d793797f6f5e25dae5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddad36ae0ddf437f9f8c6f86a538b26a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34477 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "85ac5a425d1d44d793797f6f5e25dae5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42011 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4b5b67825d8469fa032a2b34f32da06" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40465 } health_report { overall_health: UNKNOWN } } }
I20260929 19:00:54.330932 2058 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260929 19:00:54.362766 2008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:54.360154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55766 (local address 127.0.98.2:42011)
0929 19:00:54.360217 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0929 19:00:54.360219 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:00:54.360230 (+ 11us) server_negotiation.cc:407] Connection header received
0929 19:00:54.360279 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:00:54.360280 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:00:54.360294 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:00:54.360347 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 19:00:54.360490 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:54.360982 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:54.362068 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:54.362238 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:54.362289 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:54.362485 (+ 196us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 19:00:54.362567 (+ 82us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 19:00:54.362619 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0929 19:00:54.362701 (+ 82us) server_negotiation.cc:299] Negotiation successful
0929 19:00:54.362715 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260929 19:00:54.362823 2048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:54.360059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42011 (local address 127.0.0.1:55766)
0929 19:00:54.360132 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0929 19:00:54.360145 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 19:00:54.360198 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:00:54.360369 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:00:54.360370 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:00:54.360372 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 19:00:54.360427 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:00:54.360429 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:00:54.361056 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:00:54.361057 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:00:54.361985 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:00:54.361987 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:00:54.362326 (+ 339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:00:54.362327 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:00:54.362381 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:54.362402 (+ 21us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 19:00:54.362636 (+ 234us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 19:00:54.362643 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 19:00:54.362699 (+ 56us) client_negotiation.cc:240] Negotiation successful
0929 19:00:54.362747 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260929 19:00:54.363829 1661 raft_consensus.cc:1261] T b7f270fb6efd4a56b0fd4f362e925111 P ddad36ae0ddf437f9f8c6f86a538b26a [term 1 FOLLOWER]: Refusing update from remote peer 85ac5a425d1d44d793797f6f5e25dae5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 19:00:54.363893 1941 raft_consensus.cc:1261] T b7f270fb6efd4a56b0fd4f362e925111 P a4b5b67825d8469fa032a2b34f32da06 [term 1 FOLLOWER]: Refusing update from remote peer 85ac5a425d1d44d793797f6f5e25dae5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 19:00:54.364068 2056 consensus_queue.cc:1035] T b7f270fb6efd4a56b0fd4f362e925111 P 85ac5a425d1d44d793797f6f5e25dae5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddad36ae0ddf437f9f8c6f86a538b26a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34477 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 19:00:54.364207 2018 consensus_queue.cc:1035] T b7f270fb6efd4a56b0fd4f362e925111 P 85ac5a425d1d44d793797f6f5e25dae5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4b5b67825d8469fa032a2b34f32da06" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 19:00:54.364719 2017 tablet_replica.cc:439] T b7f270fb6efd4a56b0fd4f362e925111 P ddad36ae0ddf437f9f8c6f86a538b26a: TxnStatusTablet state changed. Reason: New leader 85ac5a425d1d44d793797f6f5e25dae5. Latest consensus state: current_term: 1 leader_uuid: "85ac5a425d1d44d793797f6f5e25dae5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddad36ae0ddf437f9f8c6f86a538b26a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34477 } } peers { permanent_uuid: "85ac5a425d1d44d793797f6f5e25dae5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42011 } } peers { permanent_uuid: "a4b5b67825d8469fa032a2b34f32da06" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40465 } } }
I20260929 19:00:54.364846 2017 tablet_replica.cc:442] T b7f270fb6efd4a56b0fd4f362e925111 P ddad36ae0ddf437f9f8c6f86a538b26a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 19:00:54.364854 2028 tablet_replica.cc:439] T b7f270fb6efd4a56b0fd4f362e925111 P a4b5b67825d8469fa032a2b34f32da06: TxnStatusTablet state changed. Reason: New leader 85ac5a425d1d44d793797f6f5e25dae5. Latest consensus state: current_term: 1 leader_uuid: "85ac5a425d1d44d793797f6f5e25dae5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddad36ae0ddf437f9f8c6f86a538b26a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34477 } } peers { permanent_uuid: "85ac5a425d1d44d793797f6f5e25dae5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42011 } } peers { permanent_uuid: "a4b5b67825d8469fa032a2b34f32da06" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40465 } } }
I20260929 19:00:54.364944 2028 tablet_replica.cc:442] T b7f270fb6efd4a56b0fd4f362e925111 P a4b5b67825d8469fa032a2b34f32da06: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 19:00:54.365782 2062 mvcc.cc:205] Tried to move back new op lower bound from 7334741829072543744 to 7334741828937338880. Current Snapshot: MvccSnapshot[applied={T|T < 7334741829072543744}]
I20260929 19:00:54.365947 2056 tablet_replica.cc:439] T b7f270fb6efd4a56b0fd4f362e925111 P 85ac5a425d1d44d793797f6f5e25dae5: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "85ac5a425d1d44d793797f6f5e25dae5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddad36ae0ddf437f9f8c6f86a538b26a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34477 } } peers { permanent_uuid: "85ac5a425d1d44d793797f6f5e25dae5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42011 } } peers { permanent_uuid: "a4b5b67825d8469fa032a2b34f32da06" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40465 } } }
I20260929 19:00:54.365942 2017 tablet_replica.cc:439] T b7f270fb6efd4a56b0fd4f362e925111 P ddad36ae0ddf437f9f8c6f86a538b26a: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "85ac5a425d1d44d793797f6f5e25dae5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddad36ae0ddf437f9f8c6f86a538b26a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34477 } } peers { permanent_uuid: "85ac5a425d1d44d793797f6f5e25dae5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42011 } } peers { permanent_uuid: "a4b5b67825d8469fa032a2b34f32da06" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40465 } } }
I20260929 19:00:54.366021 2017 tablet_replica.cc:442] T b7f270fb6efd4a56b0fd4f362e925111 P ddad36ae0ddf437f9f8c6f86a538b26a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 19:00:54.366021 2056 tablet_replica.cc:442] T b7f270fb6efd4a56b0fd4f362e925111 P 85ac5a425d1d44d793797f6f5e25dae5: This TxnStatusTablet replica's current role is: LEADER
I20260929 19:00:54.366266 2018 tablet_replica.cc:439] T b7f270fb6efd4a56b0fd4f362e925111 P 85ac5a425d1d44d793797f6f5e25dae5: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "85ac5a425d1d44d793797f6f5e25dae5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddad36ae0ddf437f9f8c6f86a538b26a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34477 } } peers { permanent_uuid: "85ac5a425d1d44d793797f6f5e25dae5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42011 } } peers { permanent_uuid: "a4b5b67825d8469fa032a2b34f32da06" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40465 } } }
I20260929 19:00:54.366336 2018 tablet_replica.cc:442] T b7f270fb6efd4a56b0fd4f362e925111 P 85ac5a425d1d44d793797f6f5e25dae5: This TxnStatusTablet replica's current role is: LEADER
I20260929 19:00:54.366659 2028 tablet_replica.cc:439] T b7f270fb6efd4a56b0fd4f362e925111 P a4b5b67825d8469fa032a2b34f32da06: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "85ac5a425d1d44d793797f6f5e25dae5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddad36ae0ddf437f9f8c6f86a538b26a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34477 } } peers { permanent_uuid: "85ac5a425d1d44d793797f6f5e25dae5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42011 } } peers { permanent_uuid: "a4b5b67825d8469fa032a2b34f32da06" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40465 } } }
I20260929 19:00:54.366736 2028 tablet_replica.cc:442] T b7f270fb6efd4a56b0fd4f362e925111 P a4b5b67825d8469fa032a2b34f32da06: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 19:00:54.371033 2055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:54.368494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36102 (local address 127.0.98.1:34477)
0929 19:00:54.368538 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0929 19:00:54.368540 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:00:54.368586 (+ 46us) server_negotiation.cc:407] Connection header received
0929 19:00:54.368670 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:00:54.368670 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:00:54.368689 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:00:54.368758 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 19:00:54.368893 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:54.369599 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:54.370282 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:54.370585 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:54.370654 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:54.370757 (+ 103us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 19:00:54.370861 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 19:00:54.370923 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0929 19:00:54.370946 (+ 23us) server_negotiation.cc:299] Negotiation successful
0929 19:00:54.370965 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260929 19:00:54.375952 2073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:54.373279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42011 (local address 127.0.98.1:43639)
0929 19:00:54.373424 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0929 19:00:54.373435 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 19:00:54.373492 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:00:54.373601 (+ 109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:00:54.373602 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:00:54.373603 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 19:00:54.373653 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:00:54.373654 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:00:54.374171 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:00:54.374172 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:00:54.375081 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:00:54.375083 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:00:54.375548 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:00:54.375549 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:00:54.375614 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:54.375627 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 19:00:54.375814 (+ 187us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 19:00:54.375816 (+ 2us) client_negotiation.cc:769] Sending connection context
0929 19:00:54.375861 (+ 45us) client_negotiation.cc:240] Negotiation successful
0929 19:00:54.375896 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":79,"threads_started":1}
I20260929 19:00:54.375972 2008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:54.373369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43639 (local address 127.0.98.2:42011)
0929 19:00:54.373418 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0929 19:00:54.373420 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:00:54.373484 (+ 64us) server_negotiation.cc:407] Connection header received
0929 19:00:54.373545 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:00:54.373546 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:00:54.373552 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:00:54.373589 (+ 37us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 19:00:54.373706 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:54.374106 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:54.375190 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:54.375483 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:54.375530 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:54.375710 (+ 180us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 19:00:54.375731 (+ 21us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 19:00:54.375788 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0929 19:00:54.375888 (+ 100us) server_negotiation.cc:299] Negotiation successful
0929 19:00:54.375909 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260929 19:00:54.378088 1801 raft_consensus.cc:1261] T 2848f7f03a6d4f169f8f506b1a924014 P 85ac5a425d1d44d793797f6f5e25dae5 [term 1 FOLLOWER]: Refusing update from remote peer ddad36ae0ddf437f9f8c6f86a538b26a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 19:00:54.378152 1941 raft_consensus.cc:1261] T 2848f7f03a6d4f169f8f506b1a924014 P a4b5b67825d8469fa032a2b34f32da06 [term 1 FOLLOWER]: Refusing update from remote peer ddad36ae0ddf437f9f8c6f86a538b26a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 19:00:54.378304 2041 consensus_queue.cc:1035] T 2848f7f03a6d4f169f8f506b1a924014 P ddad36ae0ddf437f9f8c6f86a538b26a [LEADER]: Connected to new peer: Peer: permanent_uuid: "85ac5a425d1d44d793797f6f5e25dae5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 19:00:54.378437 2017 consensus_queue.cc:1035] T 2848f7f03a6d4f169f8f506b1a924014 P ddad36ae0ddf437f9f8c6f86a538b26a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4b5b67825d8469fa032a2b34f32da06" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 19:00:54.379989 2064 mvcc.cc:205] Tried to move back new op lower bound from 7334741829131022336 to 7334741827516653568. Current Snapshot: MvccSnapshot[applied={T|T < 7334741829131022336}]
I20260929 19:00:54.386865 2079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:54.383641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34477 (local address 127.0.98.2:60061)
0929 19:00:54.383808 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0929 19:00:54.383818 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 19:00:54.383864 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:00:54.383981 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:00:54.383982 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:00:54.383983 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 19:00:54.384030 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:00:54.384032 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:00:54.384853 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:00:54.384854 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:00:54.386111 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:00:54.386113 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:00:54.386550 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:00:54.386551 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:00:54.386591 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:54.386601 (+ 10us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 19:00:54.386747 (+ 146us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 19:00:54.386748 (+ 1us) client_negotiation.cc:769] Sending connection context
0929 19:00:54.386781 (+ 33us) client_negotiation.cc:240] Negotiation successful
0929 19:00:54.386810 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":79,"threads_started":1}
I20260929 19:00:54.386862 2055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:54.383819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60061 (local address 127.0.98.1:34477)
0929 19:00:54.383858 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0929 19:00:54.383859 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 19:00:54.383867 (+ 8us) server_negotiation.cc:407] Connection header received
0929 19:00:54.383911 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:00:54.383912 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:00:54.383920 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:00:54.383980 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 19:00:54.384066 (+ 86us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:54.384758 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:54.386180 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:54.386479 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:54.386547 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:54.386664 (+ 117us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 19:00:54.386693 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 19:00:54.386741 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0929 19:00:54.386783 (+ 42us) server_negotiation.cc:299] Negotiation successful
0929 19:00:54.386801 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260929 19:00:54.395569 1504 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44554:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 19:00:54.395653 1504 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44554:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 19:00:54.396154 1504 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4005b06255574d27844b7b7b093be143: Sending DeleteTablet for 3 replicas of tablet 2848f7f03a6d4f169f8f506b1a924014
I20260929 19:00:54.396401 1781 tablet_service.cc:1514] Processing DeleteTablet for tablet 2848f7f03a6d4f169f8f506b1a924014 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:00:54 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55752
I20260929 19:00:54.396426 1641 tablet_service.cc:1514] Processing DeleteTablet for tablet 2848f7f03a6d4f169f8f506b1a924014 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:00:54 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36098
I20260929 19:00:54.396682 1921 tablet_service.cc:1514] Processing DeleteTablet for tablet 2848f7f03a6d4f169f8f506b1a924014 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:00:54 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48198
I20260929 19:00:54.396900 2085 tablet_replica.cc:330] T 2848f7f03a6d4f169f8f506b1a924014 P a4b5b67825d8469fa032a2b34f32da06: stopping tablet replica
I20260929 19:00:54.396997 2085 raft_consensus.cc:2229] T 2848f7f03a6d4f169f8f506b1a924014 P a4b5b67825d8469fa032a2b34f32da06 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 19:00:54.397074 2085 raft_consensus.cc:2258] T 2848f7f03a6d4f169f8f506b1a924014 P a4b5b67825d8469fa032a2b34f32da06 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 19:00:54.397316 2085 ts_tablet_manager.cc:1905] T 2848f7f03a6d4f169f8f506b1a924014 P a4b5b67825d8469fa032a2b34f32da06: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 19:00:54.397445 2084 tablet_replica.cc:330] T 2848f7f03a6d4f169f8f506b1a924014 P ddad36ae0ddf437f9f8c6f86a538b26a: stopping tablet replica
I20260929 19:00:54.397506 2084 raft_consensus.cc:2229] T 2848f7f03a6d4f169f8f506b1a924014 P ddad36ae0ddf437f9f8c6f86a538b26a [term 1 LEADER]: Raft consensus shutting down.
I20260929 19:00:54.397581 2084 raft_consensus.cc:2258] T 2848f7f03a6d4f169f8f506b1a924014 P ddad36ae0ddf437f9f8c6f86a538b26a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 19:00:54.397832 2084 ts_tablet_manager.cc:1905] T 2848f7f03a6d4f169f8f506b1a924014 P ddad36ae0ddf437f9f8c6f86a538b26a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 19:00:54.399328 2085 ts_tablet_manager.cc:1918] T 2848f7f03a6d4f169f8f506b1a924014 P a4b5b67825d8469fa032a2b34f32da06: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 19:00:54.399439 2085 log.cc:1196] T 2848f7f03a6d4f169f8f506b1a924014 P a4b5b67825d8469fa032a2b34f32da06: Deleting WAL directory at /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708447806542-392-0/minicluster-data/ts-2/wal/wals/2848f7f03a6d4f169f8f506b1a924014
I20260929 19:00:54.399524 2083 tablet_replica.cc:330] T 2848f7f03a6d4f169f8f506b1a924014 P 85ac5a425d1d44d793797f6f5e25dae5: stopping tablet replica
I20260929 19:00:54.399587 2083 raft_consensus.cc:2229] T 2848f7f03a6d4f169f8f506b1a924014 P 85ac5a425d1d44d793797f6f5e25dae5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 19:00:54.399629 2083 raft_consensus.cc:2258] T 2848f7f03a6d4f169f8f506b1a924014 P 85ac5a425d1d44d793797f6f5e25dae5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 19:00:54.399835 2083 ts_tablet_manager.cc:1905] T 2848f7f03a6d4f169f8f506b1a924014 P 85ac5a425d1d44d793797f6f5e25dae5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 19:00:54.399931 2085 ts_tablet_manager.cc:1939] T 2848f7f03a6d4f169f8f506b1a924014 P a4b5b67825d8469fa032a2b34f32da06: Deleting consensus metadata
I20260929 19:00:54.400247 1488 catalog_manager.cc:4841] TS a4b5b67825d8469fa032a2b34f32da06 (127.0.98.3:40465): tablet 2848f7f03a6d4f169f8f506b1a924014 (table test-table [id=77b84003350d49f68f29dac77eb9fd43]) successfully deleted
I20260929 19:00:54.401115 2084 ts_tablet_manager.cc:1918] T 2848f7f03a6d4f169f8f506b1a924014 P ddad36ae0ddf437f9f8c6f86a538b26a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 19:00:54.401226 2084 log.cc:1196] T 2848f7f03a6d4f169f8f506b1a924014 P ddad36ae0ddf437f9f8c6f86a538b26a: Deleting WAL directory at /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708447806542-392-0/minicluster-data/ts-0/wal/wals/2848f7f03a6d4f169f8f506b1a924014
I20260929 19:00:54.401304 2083 ts_tablet_manager.cc:1918] T 2848f7f03a6d4f169f8f506b1a924014 P 85ac5a425d1d44d793797f6f5e25dae5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 19:00:54.401367 2083 log.cc:1196] T 2848f7f03a6d4f169f8f506b1a924014 P 85ac5a425d1d44d793797f6f5e25dae5: Deleting WAL directory at /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790708447806542-392-0/minicluster-data/ts-1/wal/wals/2848f7f03a6d4f169f8f506b1a924014
I20260929 19:00:54.401589 2083 ts_tablet_manager.cc:1939] T 2848f7f03a6d4f169f8f506b1a924014 P 85ac5a425d1d44d793797f6f5e25dae5: Deleting consensus metadata
I20260929 19:00:54.401655 2084 ts_tablet_manager.cc:1939] T 2848f7f03a6d4f169f8f506b1a924014 P ddad36ae0ddf437f9f8c6f86a538b26a: Deleting consensus metadata
I20260929 19:00:54.401863 1489 catalog_manager.cc:4841] TS 85ac5a425d1d44d793797f6f5e25dae5 (127.0.98.2:42011): tablet 2848f7f03a6d4f169f8f506b1a924014 (table test-table [id=77b84003350d49f68f29dac77eb9fd43]) successfully deleted
I20260929 19:00:54.401944 1489 catalog_manager.cc:4841] TS ddad36ae0ddf437f9f8c6f86a538b26a (127.0.98.1:34477): tablet 2848f7f03a6d4f169f8f506b1a924014 (table test-table [id=77b84003350d49f68f29dac77eb9fd43]) successfully deleted
Sep 29 19:00:54 dist-test-slave-lqkv krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708453, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260929 19:00:54.415407 2055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:54.405070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36112 (local address 127.0.98.1:34477)
0929 19:00:54.405129 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0929 19:00:54.405131 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:00:54.405265 (+ 134us) server_negotiation.cc:407] Connection header received
0929 19:00:54.405321 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:00:54.405322 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:00:54.405332 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:00:54.405395 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:00:54.406077 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:54.406793 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:54.407895 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:54.408167 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:54.408239 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:54.412008 (+ 3769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:00:54.412015 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:00:54.412015 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:00:54.412022 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:00:54.414080 (+ 2058us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:00:54.414491 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:00:54.414494 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:00:54.414495 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:00:54.414569 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:00:54.414794 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:00:54.414795 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:00:54.414796 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:00:54.414946 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:00:54.415129 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0929 19:00:54.415221 (+ 92us) server_negotiation.cc:299] Negotiation successful
0929 19:00:54.415297 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
W20260929 19:00:54.415724 1701 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36112
I20260929 19:00:54.424036 2049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:54.416589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44576 (local address 127.0.98.62:39367)
0929 19:00:54.416639 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0929 19:00:54.416641 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:00:54.416816 (+ 175us) server_negotiation.cc:407] Connection header received
0929 19:00:54.416916 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:00:54.416917 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:00:54.416927 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:00:54.416956 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:00:54.417618 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:54.418330 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:54.419319 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:54.419609 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:54.419654 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:54.421069 (+ 1415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:00:54.421070 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:00:54.421070 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:00:54.421077 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:00:54.422739 (+ 1662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:00:54.423145 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:00:54.423146 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:00:54.423146 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:00:54.423200 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:00:54.423472 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:00:54.423473 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:00:54.423474 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:00:54.423594 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:00:54.423669 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 19:00:54.423873 (+ 204us) server_negotiation.cc:299] Negotiation successful
0929 19:00:54.423952 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
W20260929 19:00:54.424616 1504 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44576
I20260929 19:00:54.425283 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUbnLHE/build/release/bin/kudu with pid 1576
I20260929 19:00:54.431167 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUbnLHE/build/release/bin/kudu with pid 1716
I20260929 19:00:54.436506 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUbnLHE/build/release/bin/kudu with pid 1856
I20260929 19:00:54.441769 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUbnLHE/build/release/bin/kudu with pid 1472
2026-09-29T19:00:54Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2904 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708447806542-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 19:00:54 dist-test-slave-lqkv krb5kdc[2092](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 19:00:54 dist-test-slave-lqkv krb5kdc[2092](info): set up 2 sockets
Sep 29 19:00:54 dist-test-slave-lqkv krb5kdc[2092](info): commencing operation
krb5kdc: starting...
W20260929 19:00:55.468530 392 mini_kdc.cc:121] Time spent starting KDC: real 1.006s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708447806542-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708447806542-392-0/krb5kdc/test-user.keytab.
Sep 29 19:00:55 dist-test-slave-lqkv krb5kdc[2092](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708455, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T19:00:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T19:00:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708447806542-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708447806542-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 19:00:55.579768 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUbnLHE/build/release/bin/kudu
/tmp/dist-test-taskUbnLHE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708447806542-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708447806542-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708447806542-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708447806542-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41481
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708447806542-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41481
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708447806542-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708447806542-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708447806542-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:00:55.657712 2105 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:00:55.657939 2105 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:00:55.657977 2105 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:00:55.659080 2105 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 19:00:55.659148 2105 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260929 19:00:55.659171 2105 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:00:55.659195 2105 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:00:55.659224 2105 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 19:00:55.659250 2105 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 19:00:55.660691 2105 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708447806542-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708447806542-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41481
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708447806542-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41481
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708447806542-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2105
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708447806542-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 18:09:13 UTC on ccycloud.quasar-hpkalf.root.comops.site
build id 4680913
I20260929 19:00:55.660983 2105 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:00:55.661265 2105 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:00:55.666123 2110 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:00:55.666167 2111 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:00:55.666379 2113 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:00:55.667317 2105 server_base.cc:1034] running on GCE node
I20260929 19:00:55.667591 2105 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:00:55.667929 2105 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:00:55.669087 2105 hybrid_clock.cc:648] HybridClock initialized: now 1790708455669068 us; error 35 us; skew 500 ppm
Sep 29 19:00:55 dist-test-slave-lqkv krb5kdc[2092](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708455, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:00:55.672322 2105 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 19:00:55.672793 2105 webserver.cc:460] Webserver started at http://127.0.98.62:33159/ using document root <none> and password file <none>
I20260929 19:00:55.673077 2105 fs_manager.cc:359] Metadata directory not provided
I20260929 19:00:55.673147 2105 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:00:55.673292 2105 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:00:55.674335 2105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708447806542-392-0/minicluster-data/master-0/data/instance:
uuid: "624b000e6c514bc39d63e63ee983eca5"
format_stamp: "Formatted at 2026-09-29 19:00:55 on dist-test-slave-lqkv"
server_key: "1cce36da39960ce72e77d90eabafb61f"
server_key_iv: "991bbf6be9e8271074f7caaf4c5b8419"
server_key_version: "encryptionkey@0"
I20260929 19:00:55.674707 2105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708447806542-392-0/minicluster-data/master-0/wal/instance:
uuid: "624b000e6c514bc39d63e63ee983eca5"
format_stamp: "Formatted at 2026-09-29 19:00:55 on dist-test-slave-lqkv"
server_key: "1cce36da39960ce72e77d90eabafb61f"
server_key_iv: "991bbf6be9e8271074f7caaf4c5b8419"
server_key_version: "encryptionkey@0"
I20260929 19:00:55.675973 2105 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 19:00:55.676638 2120 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:00:55.676797 2105 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 19:00:55.676918 2105 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708447806542-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708447806542-392-0/minicluster-data/master-0/wal
uuid: "624b000e6c514bc39d63e63ee983eca5"
format_stamp: "Formatted at 2026-09-29 19:00:55 on dist-test-slave-lqkv"
server_key: "1cce36da39960ce72e77d90eabafb61f"
server_key_iv: "991bbf6be9e8271074f7caaf4c5b8419"
server_key_version: "encryptionkey@0"
I20260929 19:00:55.677003 2105 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708447806542-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708447806542-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708447806542-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:00:55.703028 2105 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:00:55.705111 2105 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:00:55.705273 2105 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:00:55.709906 2105 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41481
I20260929 19:00:55.709961 2182 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41481 every 8 connection(s)
I20260929 19:00:55.710314 2105 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708447806542-392-0/minicluster-data/master-0/data/info.pb
I20260929 19:00:55.711047 2183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:00:55.713138 2183 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 624b000e6c514bc39d63e63ee983eca5: Bootstrap starting.
I20260929 19:00:55.713644 2183 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 624b000e6c514bc39d63e63ee983eca5: Neither blocks nor log segments found. Creating new log.
I20260929 19:00:55.713878 2183 log.cc:824] T 00000000000000000000000000000000 P 624b000e6c514bc39d63e63ee983eca5: Log is configured to *not* fsync() on all Append() calls
I20260929 19:00:55.714377 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUbnLHE/build/release/bin/kudu as pid 2105
I20260929 19:00:55.714480 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708447806542-392-0/minicluster-data/master-0/wal/instance
I20260929 19:00:55.714519 2183 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 624b000e6c514bc39d63e63ee983eca5: No bootstrap required, opened a new log
I20260929 19:00:55.714668 392 external_mini_cluster.cc:1428] Setting key 36e41cf013bc26cd045df32481859c35
I20260929 19:00:55.717283 2183 raft_consensus.cc:348] T 00000000000000000000000000000000 P 624b000e6c514bc39d63e63ee983eca5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "624b000e6c514bc39d63e63ee983eca5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41481 } }
I20260929 19:00:55.717409 2183 raft_consensus.cc:374] T 00000000000000000000000000000000 P 624b000e6c514bc39d63e63ee983eca5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:00:55.717432 2183 raft_consensus.cc:729] T 00000000000000000000000000000000 P 624b000e6c514bc39d63e63ee983eca5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 624b000e6c514bc39d63e63ee983eca5, State: Initialized, Role: FOLLOWER
I20260929 19:00:55.717561 2183 consensus_queue.cc:260] T 00000000000000000000000000000000 P 624b000e6c514bc39d63e63ee983eca5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "624b000e6c514bc39d63e63ee983eca5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41481 } }
I20260929 19:00:55.717631 2183 raft_consensus.cc:388] T 00000000000000000000000000000000 P 624b000e6c514bc39d63e63ee983eca5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 19:00:55.717671 2183 raft_consensus.cc:482] T 00000000000000000000000000000000 P 624b000e6c514bc39d63e63ee983eca5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 19:00:55.717718 2183 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 624b000e6c514bc39d63e63ee983eca5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:00:55.718407 2183 raft_consensus.cc:504] T 00000000000000000000000000000000 P 624b000e6c514bc39d63e63ee983eca5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "624b000e6c514bc39d63e63ee983eca5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41481 } }
I20260929 19:00:55.718511 2183 leader_election.cc:302] T 00000000000000000000000000000000 P 624b000e6c514bc39d63e63ee983eca5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 624b000e6c514bc39d63e63ee983eca5; no voters:
I20260929 19:00:55.718648 2183 leader_election.cc:288] T 00000000000000000000000000000000 P 624b000e6c514bc39d63e63ee983eca5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 19:00:55.718712 2188 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 624b000e6c514bc39d63e63ee983eca5 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:00:55.718845 2188 raft_consensus.cc:686] T 00000000000000000000000000000000 P 624b000e6c514bc39d63e63ee983eca5 [term 1 LEADER]: Becoming Leader. State: Replica: 624b000e6c514bc39d63e63ee983eca5, State: Running, Role: LEADER
I20260929 19:00:55.718866 2183 sys_catalog.cc:564] T 00000000000000000000000000000000 P 624b000e6c514bc39d63e63ee983eca5 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 19:00:55.718937 2188 consensus_queue.cc:237] T 00000000000000000000000000000000 P 624b000e6c514bc39d63e63ee983eca5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "624b000e6c514bc39d63e63ee983eca5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41481 } }
I20260929 19:00:55.719215 2190 sys_catalog.cc:455] T 00000000000000000000000000000000 P 624b000e6c514bc39d63e63ee983eca5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "624b000e6c514bc39d63e63ee983eca5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "624b000e6c514bc39d63e63ee983eca5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41481 } } }
I20260929 19:00:55.719203 2188 sys_catalog.cc:455] T 00000000000000000000000000000000 P 624b000e6c514bc39d63e63ee983eca5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 624b000e6c514bc39d63e63ee983eca5. Latest consensus state: current_term: 1 leader_uuid: "624b000e6c514bc39d63e63ee983eca5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "624b000e6c514bc39d63e63ee983eca5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41481 } } }
I20260929 19:00:55.719375 2188 sys_catalog.cc:458] T 00000000000000000000000000000000 P 624b000e6c514bc39d63e63ee983eca5 [sys.catalog]: This master's current role is: LEADER
I20260929 19:00:55.719328 2190 sys_catalog.cc:458] T 00000000000000000000000000000000 P 624b000e6c514bc39d63e63ee983eca5 [sys.catalog]: This master's current role is: LEADER
I20260929 19:00:55.719592 2196 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 19:00:55.720110 2196 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 19:00:55.721829 2196 catalog_manager.cc:1347] Generated new cluster ID: 1849f9b7671b49fba0f0bb415ed9d64c
I20260929 19:00:55.721885 2196 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 29 19:00:55 dist-test-slave-lqkv krb5kdc[2092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708455, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 19:00:55.726267 2187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:55.715192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42818 (local address 127.0.98.62:41481)
0929 19:00:55.715379 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0929 19:00:55.715386 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 19:00:55.715411 (+ 25us) server_negotiation.cc:407] Connection header received
0929 19:00:55.715834 (+ 423us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:00:55.715840 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:00:55.715886 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:00:55.715992 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:00:55.716836 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:55.717914 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:55.718969 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:55.719662 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:55.719754 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:55.722803 (+ 3049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:00:55.722807 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:00:55.722807 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:00:55.722818 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:00:55.724674 (+ 1856us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:00:55.725101 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:00:55.725103 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:00:55.725104 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:00:55.725197 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:00:55.725408 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:00:55.725409 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:00:55.725409 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:00:55.725670 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:00:55.725810 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0929 19:00:55.725955 (+ 145us) server_negotiation.cc:299] Negotiation successful
0929 19:00:55.726073 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":90,"threads_started":1}
I20260929 19:00:55.746497 2196 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 19:00:55.747244 2196 catalog_manager.cc:1504] Loading token signing keys...
I20260929 19:00:55.752167 2196 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 624b000e6c514bc39d63e63ee983eca5: Generated new TSK 0
I20260929 19:00:55.752351 2196 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708447806542-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708447806542-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260929 19:00:55.775791 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUbnLHE/build/release/bin/kudu
/tmp/dist-test-taskUbnLHE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708447806542-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708447806542-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708447806542-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708447806542-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41481
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708447806542-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708447806542-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708447806542-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708447806542-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:00:55.853826 2209 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:00:55.854027 2209 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:00:55.854049 2209 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:00:55.855103 2209 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:00:55.855139 2209 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:00:55.855158 2209 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260929 19:00:55.855175 2209 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 19:00:55.855192 2209 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 19:00:55.856523 2209 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708447806542-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708447806542-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708447806542-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708447806542-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:41481
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2209
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708447806542-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 18:09:13 UTC on ccycloud.quasar-hpkalf.root.comops.site
build id 4680913
I20260929 19:00:55.856695 2209 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:00:55.856922 2209 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:00:55.861603 2217 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:00:55.861685 2215 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:00:55.861723 2209 server_base.cc:1034] running on GCE node
W20260929 19:00:55.861873 2214 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:00:55.862079 2209 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:00:55.862262 2209 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:00:55.863443 2209 hybrid_clock.cc:648] HybridClock initialized: now 1790708455863431 us; error 42 us; skew 500 ppm
Sep 29 19:00:55 dist-test-slave-lqkv krb5kdc[2092](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708455, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:00:55.865839 2209 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 19:00:55.866293 2209 webserver.cc:460] Webserver started at http://127.0.98.1:39475/ using document root <none> and password file <none>
I20260929 19:00:55.866519 2209 fs_manager.cc:359] Metadata directory not provided
I20260929 19:00:55.866564 2209 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:00:55.866700 2209 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:00:55.867690 2209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708447806542-392-0/minicluster-data/ts-0/data/instance:
uuid: "3411d1f14ac94407b9fe8e491584772b"
format_stamp: "Formatted at 2026-09-29 19:00:55 on dist-test-slave-lqkv"
server_key: "05c2a8c68ed60100655f1e5b8acdfa2f"
server_key_iv: "e566e8ced1c7ed873d72733d9522a1f8"
server_key_version: "encryptionkey@0"
I20260929 19:00:55.868088 2209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708447806542-392-0/minicluster-data/ts-0/wal/instance:
uuid: "3411d1f14ac94407b9fe8e491584772b"
format_stamp: "Formatted at 2026-09-29 19:00:55 on dist-test-slave-lqkv"
server_key: "05c2a8c68ed60100655f1e5b8acdfa2f"
server_key_iv: "e566e8ced1c7ed873d72733d9522a1f8"
server_key_version: "encryptionkey@0"
I20260929 19:00:55.869431 2209 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 19:00:55.870118 2224 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:00:55.870297 2209 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 19:00:55.870380 2209 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708447806542-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708447806542-392-0/minicluster-data/ts-0/wal
uuid: "3411d1f14ac94407b9fe8e491584772b"
format_stamp: "Formatted at 2026-09-29 19:00:55 on dist-test-slave-lqkv"
server_key: "05c2a8c68ed60100655f1e5b8acdfa2f"
server_key_iv: "e566e8ced1c7ed873d72733d9522a1f8"
server_key_version: "encryptionkey@0"
I20260929 19:00:55.870448 2209 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708447806542-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708447806542-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708447806542-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:00:55.900106 2209 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:00:55.902177 2209 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:00:55.902292 2209 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:00:55.902741 2209 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:00:55.902784 2209 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:00:55.902806 2209 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:00:55.902818 2209 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:00:55.909633 2209 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42221
I20260929 19:00:55.909708 2339 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42221 every 8 connection(s)
Sep 29 19:00:55 dist-test-slave-lqkv krb5kdc[2092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708455, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 19:00:55.910033 2209 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708447806542-392-0/minicluster-data/ts-0/data/info.pb
I20260929 19:00:55.910425 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUbnLHE/build/release/bin/kudu as pid 2209
I20260929 19:00:55.910519 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708447806542-392-0/minicluster-data/ts-0/wal/instance
I20260929 19:00:55.910699 392 external_mini_cluster.cc:1428] Setting key 2fe882eca4fc2b2a4f753471a0e7d005
I20260929 19:00:55.913806 2187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:55.903234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38063 (local address 127.0.98.62:41481)
0929 19:00:55.903312 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0929 19:00:55.903315 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 19:00:55.903485 (+ 170us) server_negotiation.cc:407] Connection header received
0929 19:00:55.903956 (+ 471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:00:55.903958 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:00:55.903970 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:00:55.904051 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:00:55.904928 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:55.905757 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:55.907395 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:55.907765 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:55.907845 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:55.910559 (+ 2714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:00:55.910560 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:00:55.910561 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:00:55.910568 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:00:55.912403 (+ 1835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:00:55.912850 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:00:55.912851 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:00:55.912852 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:00:55.912919 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:00:55.913173 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:00:55.913173 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:00:55.913174 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:00:55.913290 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:00:55.913366 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 19:00:55.913629 (+ 263us) server_negotiation.cc:299] Negotiation successful
0929 19:00:55.913708 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 19:00:55.913970 2233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:55.903189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41481 (local address 127.0.98.1:38063)
0929 19:00:55.903350 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0929 19:00:55.903389 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 19:00:55.903860 (+ 471us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:00:55.904140 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:00:55.904145 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:00:55.904507 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:00:55.904799 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:00:55.904806 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:00:55.905864 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:00:55.905865 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:00:55.907316 (+ 1451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:00:55.907319 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:00:55.907852 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:00:55.907853 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:00:55.907949 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:55.908466 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:00:55.908478 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:00:55.910443 (+ 1965us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:00:55.912483 (+ 2040us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:00:55.912488 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:00:55.912490 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:00:55.912750 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:00:55.912987 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:00:55.912988 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:00:55.912988 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:00:55.913100 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:00:55.913356 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:00:55.913362 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:00:55.913500 (+ 138us) client_negotiation.cc:769] Sending connection context
0929 19:00:55.913597 (+ 97us) client_negotiation.cc:240] Negotiation successful
0929 19:00:55.913746 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":93,"threads_started":1}
I20260929 19:00:55.914448 2340 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41481
I20260929 19:00:55.914536 2340 heartbeater.cc:464] Registering TS with master...
I20260929 19:00:55.914716 2340 heartbeater.cc:511] Master 127.0.98.62:41481 requested a full tablet report, sending...
I20260929 19:00:55.915226 2136 ts_manager.cc:194] Registered new tserver with Master: 3411d1f14ac94407b9fe8e491584772b (127.0.98.1:42221)
I20260929 19:00:55.916211 2136 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:38063
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260929 19:00:55.922932 2187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:55.915519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35849 (local address 127.0.98.62:41481)
0929 19:00:55.915560 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0929 19:00:55.915562 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:00:55.916578 (+ 1016us) server_negotiation.cc:407] Connection header received
0929 19:00:55.917248 (+ 670us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:00:55.917249 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:00:55.917259 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:00:55.917305 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:00:55.917649 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:55.918099 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:55.919139 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:55.919368 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:55.919420 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:55.920236 (+ 816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:00:55.920237 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:00:55.920237 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:00:55.920243 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:00:55.921679 (+ 1436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:00:55.922104 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:00:55.922105 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:00:55.922105 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:00:55.922153 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:00:55.922435 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:00:55.922436 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:00:55.922436 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:00:55.922543 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:00:55.922617 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 19:00:55.922778 (+ 161us) server_negotiation.cc:299] Negotiation successful
0929 19:00:55.922843 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260929 19:00:55.922945 2347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:55.915480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41481 (local address 127.0.98.1:35849)
0929 19:00:55.916498 (+ 1018us) negotiation.cc:107] Waiting for socket to connect
0929 19:00:55.916511 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 19:00:55.916579 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:00:55.917314 (+ 735us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:00:55.917315 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:00:55.917519 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:00:55.917585 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:00:55.917586 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:00:55.918202 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:00:55.918203 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:00:55.919065 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:00:55.919068 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:00:55.919422 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:00:55.919423 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:00:55.919496 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:55.919901 (+ 405us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:00:55.919906 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:00:55.920164 (+ 258us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:00:55.921775 (+ 1611us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:00:55.921776 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:00:55.921777 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:00:55.922036 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:00:55.922278 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:00:55.922279 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:00:55.922279 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:00:55.922363 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:00:55.922609 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:00:55.922610 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:00:55.922660 (+ 50us) client_negotiation.cc:769] Sending connection context
0929 19:00:55.922745 (+ 85us) client_negotiation.cc:240] Negotiation successful
0929 19:00:55.922845 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":994,"thread_start_us":73,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708447806542-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708447806542-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260929 19:00:55.935189 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUbnLHE/build/release/bin/kudu
/tmp/dist-test-taskUbnLHE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708447806542-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708447806542-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708447806542-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708447806542-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41481
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708447806542-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708447806542-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708447806542-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708447806542-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:00:56.014181 2349 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:00:56.014374 2349 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:00:56.014398 2349 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:00:56.015537 2349 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:00:56.015578 2349 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:00:56.015599 2349 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260929 19:00:56.015610 2349 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 19:00:56.015627 2349 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 19:00:56.016986 2349 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708447806542-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708447806542-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708447806542-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708447806542-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:41481
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2349
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708447806542-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 18:09:13 UTC on ccycloud.quasar-hpkalf.root.comops.site
build id 4680913
I20260929 19:00:56.017164 2349 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:00:56.017320 2349 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:00:56.022248 2357 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:00:56.022259 2355 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:00:56.022315 2349 server_base.cc:1034] running on GCE node
W20260929 19:00:56.022260 2354 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:00:56.022639 2349 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:00:56.022909 2349 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:00:56.024091 2349 hybrid_clock.cc:648] HybridClock initialized: now 1790708456024076 us; error 23 us; skew 500 ppm
Sep 29 19:00:56 dist-test-slave-lqkv krb5kdc[2092](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708456, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:00:56.026553 2349 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 19:00:56.027004 2349 webserver.cc:460] Webserver started at http://127.0.98.2:38965/ using document root <none> and password file <none>
I20260929 19:00:56.027233 2349 fs_manager.cc:359] Metadata directory not provided
I20260929 19:00:56.027300 2349 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:00:56.027446 2349 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:00:56.028447 2349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708447806542-392-0/minicluster-data/ts-1/data/instance:
uuid: "0725f0c1e7c74171b33e880e6a6fd5cd"
format_stamp: "Formatted at 2026-09-29 19:00:56 on dist-test-slave-lqkv"
server_key: "a968b93d8e3d7d57ac407828f2f57a2d"
server_key_iv: "90b8b761c376b58801f4283f11658164"
server_key_version: "encryptionkey@0"
I20260929 19:00:56.028843 2349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708447806542-392-0/minicluster-data/ts-1/wal/instance:
uuid: "0725f0c1e7c74171b33e880e6a6fd5cd"
format_stamp: "Formatted at 2026-09-29 19:00:56 on dist-test-slave-lqkv"
server_key: "a968b93d8e3d7d57ac407828f2f57a2d"
server_key_iv: "90b8b761c376b58801f4283f11658164"
server_key_version: "encryptionkey@0"
I20260929 19:00:56.030085 2349 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 19:00:56.030845 2364 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:00:56.031028 2349 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 19:00:56.031124 2349 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708447806542-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708447806542-392-0/minicluster-data/ts-1/wal
uuid: "0725f0c1e7c74171b33e880e6a6fd5cd"
format_stamp: "Formatted at 2026-09-29 19:00:56 on dist-test-slave-lqkv"
server_key: "a968b93d8e3d7d57ac407828f2f57a2d"
server_key_iv: "90b8b761c376b58801f4283f11658164"
server_key_version: "encryptionkey@0"
I20260929 19:00:56.031219 2349 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708447806542-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708447806542-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708447806542-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:00:56.050683 2349 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:00:56.052597 2349 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:00:56.052759 2349 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:00:56.053335 2349 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:00:56.053392 2349 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:00:56.053449 2349 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:00:56.053468 2349 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 19:00:56 dist-test-slave-lqkv krb5kdc[2092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708456, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 19:00:56.060497 2349 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42581
I20260929 19:00:56.060572 2479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42581 every 8 connection(s)
I20260929 19:00:56.060945 2349 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708447806542-392-0/minicluster-data/ts-1/data/info.pb
I20260929 19:00:56.062847 2187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:56.053733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34049 (local address 127.0.98.62:41481)
0929 19:00:56.053790 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0929 19:00:56.053792 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:00:56.054046 (+ 254us) server_negotiation.cc:407] Connection header received
0929 19:00:56.054558 (+ 512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:00:56.054559 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:00:56.054571 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:00:56.054624 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:00:56.055233 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:56.055986 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:56.057323 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:56.057533 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:56.057617 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:56.060206 (+ 2589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:00:56.060208 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:00:56.060208 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:00:56.060215 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:00:56.061727 (+ 1512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:00:56.062139 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:00:56.062139 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:00:56.062140 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:00:56.062188 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:00:56.062390 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:00:56.062390 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:00:56.062390 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:00:56.062478 (+ 88us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:00:56.062521 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0929 19:00:56.062734 (+ 213us) server_negotiation.cc:299] Negotiation successful
0929 19:00:56.062788 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260929 19:00:56.063093 2373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:56.053725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41481 (local address 127.0.98.2:34049)
0929 19:00:56.053931 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0929 19:00:56.053964 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0929 19:00:56.054456 (+ 492us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:00:56.054707 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:00:56.054710 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:00:56.054942 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:00:56.055130 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:00:56.055134 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:00:56.056085 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:00:56.056087 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:00:56.057236 (+ 1149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:00:56.057239 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:00:56.057636 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:00:56.057637 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:00:56.057771 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:56.058255 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:00:56.058265 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:00:56.060098 (+ 1833us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:00:56.061786 (+ 1688us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:00:56.061790 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:00:56.061791 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:00:56.062077 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:00:56.062249 (+ 172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:00:56.062250 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:00:56.062250 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:00:56.062350 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:00:56.062525 (+ 175us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:00:56.062529 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:00:56.062638 (+ 109us) client_negotiation.cc:769] Sending connection context
0929 19:00:56.062714 (+ 76us) client_negotiation.cc:240] Negotiation successful
0929 19:00:56.062879 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":132,"threads_started":1}
I20260929 19:00:56.063601 2480 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41481
I20260929 19:00:56.063680 2480 heartbeater.cc:464] Registering TS with master...
I20260929 19:00:56.063868 2480 heartbeater.cc:511] Master 127.0.98.62:41481 requested a full tablet report, sending...
I20260929 19:00:56.064210 2137 ts_manager.cc:194] Registered new tserver with Master: 0725f0c1e7c74171b33e880e6a6fd5cd (127.0.98.2:42581)
I20260929 19:00:56.064751 2137 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:34049
I20260929 19:00:56.069428 2187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:56.064644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49549 (local address 127.0.98.62:41481)
0929 19:00:56.064683 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0929 19:00:56.064685 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:00:56.064763 (+ 78us) server_negotiation.cc:407] Connection header received
0929 19:00:56.064864 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:00:56.064865 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:00:56.064874 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:00:56.064929 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:00:56.065334 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:56.065818 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:56.066459 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:56.066622 (+ 163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:56.066668 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:56.067516 (+ 848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:00:56.067516 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:00:56.067517 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:00:56.067521 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:00:56.068576 (+ 1055us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:00:56.068911 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:00:56.068912 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:00:56.068912 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:00:56.068955 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:00:56.069072 (+ 117us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:00:56.069072 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:00:56.069072 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:00:56.069174 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:00:56.069225 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0929 19:00:56.069329 (+ 104us) server_negotiation.cc:299] Negotiation successful
0929 19:00:56.069378 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260929 19:00:56.069445 2486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:56.064562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41481 (local address 127.0.98.2:49549)
0929 19:00:56.064701 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0929 19:00:56.064712 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 19:00:56.064771 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:00:56.065059 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:00:56.065059 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:00:56.065209 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:00:56.065260 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:00:56.065261 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:00:56.065901 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:00:56.065902 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:00:56.066393 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:00:56.066394 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:00:56.066689 (+ 295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:00:56.066691 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:00:56.066771 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:56.067179 (+ 408us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:00:56.067184 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:00:56.067448 (+ 264us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:00:56.068648 (+ 1200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:00:56.068649 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:00:56.068650 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:00:56.068840 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:00:56.068994 (+ 154us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:00:56.068994 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:00:56.068994 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:00:56.069039 (+ 45us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:00:56.069233 (+ 194us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:00:56.069234 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:00:56.069267 (+ 33us) client_negotiation.cc:769] Sending connection context
0929 19:00:56.069313 (+ 46us) client_negotiation.cc:240] Negotiation successful
0929 19:00:56.069381 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":96,"threads_started":1}
I20260929 19:00:56.069749 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUbnLHE/build/release/bin/kudu as pid 2349
I20260929 19:00:56.069825 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708447806542-392-0/minicluster-data/ts-1/wal/instance
I20260929 19:00:56.069993 392 external_mini_cluster.cc:1428] Setting key 83429317a417577d866a5202d8df5007
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708447806542-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708447806542-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260929 19:00:56.093484 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUbnLHE/build/release/bin/kudu
/tmp/dist-test-taskUbnLHE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708447806542-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708447806542-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708447806542-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708447806542-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41481
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708447806542-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708447806542-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708447806542-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708447806542-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:00:56.171890 2489 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:00:56.172083 2489 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:00:56.172107 2489 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:00:56.173275 2489 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:00:56.173312 2489 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:00:56.173332 2489 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260929 19:00:56.173344 2489 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 19:00:56.173362 2489 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 19:00:56.174743 2489 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708447806542-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708447806542-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708447806542-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708447806542-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:41481
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2489
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708447806542-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 18:09:13 UTC on ccycloud.quasar-hpkalf.root.comops.site
build id 4680913
I20260929 19:00:56.174917 2489 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:00:56.175083 2489 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:00:56.179972 2494 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:00:56.179981 2495 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:00:56.180039 2497 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:00:56.180246 2489 server_base.cc:1034] running on GCE node
I20260929 19:00:56.180445 2489 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:00:56.180678 2489 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:00:56.181852 2489 hybrid_clock.cc:648] HybridClock initialized: now 1790708456181813 us; error 49 us; skew 500 ppm
Sep 29 19:00:56 dist-test-slave-lqkv krb5kdc[2092](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708456, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:00:56.184175 2489 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 19:00:56.184648 2489 webserver.cc:460] Webserver started at http://127.0.98.3:37663/ using document root <none> and password file <none>
I20260929 19:00:56.184942 2489 fs_manager.cc:359] Metadata directory not provided
I20260929 19:00:56.185011 2489 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:00:56.185161 2489 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:00:56.186190 2489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708447806542-392-0/minicluster-data/ts-2/data/instance:
uuid: "5fd1996f61cc4be2858c26c0f009154b"
format_stamp: "Formatted at 2026-09-29 19:00:56 on dist-test-slave-lqkv"
server_key: "bbfd1ecd28b27baea884c602593b4d66"
server_key_iv: "f2aacd403eead604fbfe492abb44d2f0"
server_key_version: "encryptionkey@0"
I20260929 19:00:56.186547 2489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708447806542-392-0/minicluster-data/ts-2/wal/instance:
uuid: "5fd1996f61cc4be2858c26c0f009154b"
format_stamp: "Formatted at 2026-09-29 19:00:56 on dist-test-slave-lqkv"
server_key: "bbfd1ecd28b27baea884c602593b4d66"
server_key_iv: "f2aacd403eead604fbfe492abb44d2f0"
server_key_version: "encryptionkey@0"
I20260929 19:00:56.187829 2489 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 19:00:56.188503 2504 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:00:56.188671 2489 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 19:00:56.188768 2489 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708447806542-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708447806542-392-0/minicluster-data/ts-2/wal
uuid: "5fd1996f61cc4be2858c26c0f009154b"
format_stamp: "Formatted at 2026-09-29 19:00:56 on dist-test-slave-lqkv"
server_key: "bbfd1ecd28b27baea884c602593b4d66"
server_key_iv: "f2aacd403eead604fbfe492abb44d2f0"
server_key_version: "encryptionkey@0"
I20260929 19:00:56.188880 2489 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708447806542-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708447806542-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708447806542-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:00:56.209488 2489 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:00:56.211479 2489 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:00:56.211648 2489 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:00:56.212205 2489 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:00:56.212265 2489 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:00:56.212306 2489 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:00:56.212332 2489 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 19:00:56 dist-test-slave-lqkv krb5kdc[2092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708456, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 19:00:56.219772 2489 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45245
I20260929 19:00:56.220188 2489 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708447806542-392-0/minicluster-data/ts-2/data/info.pb
I20260929 19:00:56.220575 2619 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45245 every 8 connection(s)
I20260929 19:00:56.222765 2187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:56.212745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50463 (local address 127.0.98.62:41481)
0929 19:00:56.212837 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0929 19:00:56.212840 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 19:00:56.212995 (+ 155us) server_negotiation.cc:407] Connection header received
0929 19:00:56.213456 (+ 461us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:00:56.213457 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:00:56.213471 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:00:56.213551 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:00:56.214211 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:56.214864 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:56.216062 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:56.216287 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:56.216385 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:56.219758 (+ 3373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:00:56.219759 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:00:56.219760 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:00:56.219767 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:00:56.221414 (+ 1647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:00:56.221822 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:00:56.221823 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:00:56.221824 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:00:56.221880 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:00:56.222134 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:00:56.222135 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:00:56.222135 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:00:56.222259 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:00:56.222334 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 19:00:56.222594 (+ 260us) server_negotiation.cc:299] Negotiation successful
0929 19:00:56.222670 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 19:00:56.222957 2513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:56.212674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41481 (local address 127.0.98.3:50463)
0929 19:00:56.212859 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0929 19:00:56.212898 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 19:00:56.213366 (+ 468us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:00:56.213615 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:00:56.213619 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:00:56.213865 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:00:56.214090 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:00:56.214097 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:00:56.214970 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:00:56.214971 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:00:56.215967 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:00:56.215970 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:00:56.216393 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:00:56.216394 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:00:56.216523 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:56.217099 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:00:56.217109 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:00:56.219659 (+ 2550us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:00:56.221512 (+ 1853us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:00:56.221518 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:00:56.221520 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:00:56.221746 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:00:56.221954 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:00:56.221955 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:00:56.221955 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:00:56.222068 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:00:56.222325 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:00:56.222330 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:00:56.222468 (+ 138us) client_negotiation.cc:769] Sending connection context
0929 19:00:56.222575 (+ 107us) client_negotiation.cc:240] Negotiation successful
0929 19:00:56.222738 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":90,"threads_started":1}
I20260929 19:00:56.223454 2620 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41481
I20260929 19:00:56.223536 2620 heartbeater.cc:464] Registering TS with master...
I20260929 19:00:56.223707 2620 heartbeater.cc:511] Master 127.0.98.62:41481 requested a full tablet report, sending...
I20260929 19:00:56.224108 2136 ts_manager.cc:194] Registered new tserver with Master: 5fd1996f61cc4be2858c26c0f009154b (127.0.98.3:45245)
I20260929 19:00:56.224604 2136 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:50463
I20260929 19:00:56.227825 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUbnLHE/build/release/bin/kudu as pid 2489
I20260929 19:00:56.227921 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708447806542-392-0/minicluster-data/ts-2/wal/instance
I20260929 19:00:56.228089 392 external_mini_cluster.cc:1428] Setting key 91d734e70298518482aeec287311674c
I20260929 19:00:56.228962 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 19:00:56.230058 2187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:56.224474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34033 (local address 127.0.98.62:41481)
0929 19:00:56.224507 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0929 19:00:56.224509 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:00:56.224593 (+ 84us) server_negotiation.cc:407] Connection header received
0929 19:00:56.224652 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:00:56.224652 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:00:56.224677 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:00:56.224735 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:00:56.225140 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:56.225705 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:56.226359 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:56.226661 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:56.226713 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:56.227433 (+ 720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:00:56.227434 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:00:56.227434 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:00:56.227439 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:00:56.229087 (+ 1648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:00:56.229439 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:00:56.229440 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:00:56.229440 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:00:56.229487 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:00:56.229666 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:00:56.229666 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:00:56.229667 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:00:56.229754 (+ 87us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:00:56.229798 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0929 19:00:56.229954 (+ 156us) server_negotiation.cc:299] Negotiation successful
0929 19:00:56.230005 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260929 19:00:56.230206 2626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:56.224405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41481 (local address 127.0.98.3:34033)
0929 19:00:56.224539 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0929 19:00:56.224549 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 19:00:56.224599 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:00:56.224861 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:00:56.224862 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:00:56.225015 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:00:56.225067 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:00:56.225068 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:00:56.225786 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:00:56.225787 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:00:56.226288 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:00:56.226289 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:00:56.226725 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:00:56.226725 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:00:56.226783 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:56.227100 (+ 317us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:00:56.227104 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:00:56.227379 (+ 275us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:00:56.229167 (+ 1788us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:00:56.229168 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:00:56.229168 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:00:56.229362 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:00:56.229546 (+ 184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:00:56.229547 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:00:56.229547 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:00:56.229610 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:00:56.229813 (+ 203us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:00:56.229814 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:00:56.229860 (+ 46us) client_negotiation.cc:769] Sending connection context
0929 19:00:56.229937 (+ 77us) client_negotiation.cc:240] Negotiation successful
0929 19:00:56.230050 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":76,"threads_started":1}
Sep 29 19:00:56 dist-test-slave-lqkv krb5kdc[2092](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708456, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 19:00:56 dist-test-slave-lqkv krb5kdc[2092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708456, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 19:00:56.251230 2187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:56.244668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42830 (local address 127.0.98.62:41481)
0929 19:00:56.244763 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0929 19:00:56.244765 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:00:56.244774 (+ 9us) server_negotiation.cc:407] Connection header received
0929 19:00:56.244831 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:00:56.244832 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:00:56.244843 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:00:56.244917 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:00:56.245473 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:56.245955 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:56.246674 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:56.246867 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:56.246952 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:56.249010 (+ 2058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:00:56.249011 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:00:56.249012 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:00:56.249020 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:00:56.250155 (+ 1135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:00:56.250489 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:00:56.250490 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:00:56.250491 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:00:56.250548 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:00:56.250743 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:00:56.250744 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:00:56.250744 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:00:56.250851 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:00:56.250957 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0929 19:00:56.251058 (+ 101us) server_negotiation.cc:299] Negotiation successful
0929 19:00:56.251128 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 19:00:56.252918 2136 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42830:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 19:00:56.254369 2136 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 19:00:56.260388 2638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:56.256384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42221 (local address 127.0.0.1:55212)
0929 19:00:56.256529 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0929 19:00:56.256570 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 19:00:56.256630 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:00:56.257071 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:00:56.257072 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:00:56.257075 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:00:56.257131 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:00:56.257133 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:00:56.258145 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:00:56.258146 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:00:56.259247 (+ 1101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:00:56.259249 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:00:56.260124 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:00:56.260125 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:00:56.260176 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:56.260187 (+ 11us) client_negotiation.cc:769] Sending connection context
0929 19:00:56.260267 (+ 80us) client_negotiation.cc:240] Negotiation successful
0929 19:00:56.260315 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":45,"threads_started":1}
I20260929 19:00:56.260433 2641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:56.256712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55212 (local address 127.0.98.1:42221)
0929 19:00:56.256915 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0929 19:00:56.256919 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 19:00:56.256928 (+ 9us) server_negotiation.cc:407] Connection header received
0929 19:00:56.256963 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:00:56.256966 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:00:56.256980 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:00:56.257052 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:00:56.257312 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:56.258048 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:56.259344 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:56.260006 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:56.260040 (+ 34us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:56.260078 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0929 19:00:56.260283 (+ 205us) server_negotiation.cc:299] Negotiation successful
0929 19:00:56.260334 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":80,"threads_started":1}
I20260929 19:00:56.260632 2637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:56.256406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42581 (local address 127.0.0.1:53396)
0929 19:00:56.256531 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0929 19:00:56.256570 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 19:00:56.256632 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:00:56.257057 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:00:56.257059 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:00:56.257062 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:00:56.257124 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:00:56.257126 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:00:56.258656 (+ 1530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:00:56.258658 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:00:56.259516 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:00:56.259518 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:00:56.260400 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:00:56.260401 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:00:56.260470 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:56.260475 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 19:00:56.260508 (+ 33us) client_negotiation.cc:240] Negotiation successful
0929 19:00:56.260551 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":72,"threads_started":1}
I20260929 19:00:56.260775 2640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:56.256554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45245 (local address 127.0.0.1:49880)
0929 19:00:56.256658 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0929 19:00:56.256667 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 19:00:56.256707 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:00:56.257366 (+ 659us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:00:56.257367 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:00:56.257369 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:00:56.257405 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:00:56.257406 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:00:56.258627 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:00:56.258627 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:00:56.259668 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:00:56.259670 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:00:56.260524 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:00:56.260525 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:00:56.260571 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:56.260574 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 19:00:56.260602 (+ 28us) client_negotiation.cc:240] Negotiation successful
0929 19:00:56.260644 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87,"thread_start_us":67,"threads_started":1}
I20260929 19:00:56.260938 2642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:56.256975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49880 (local address 127.0.98.3:45245)
0929 19:00:56.257198 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0929 19:00:56.257202 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 19:00:56.257210 (+ 8us) server_negotiation.cc:407] Connection header received
0929 19:00:56.257256 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:00:56.257259 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:00:56.257273 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:00:56.257482 (+ 209us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:00:56.257537 (+ 55us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:56.258538 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:56.259745 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:56.260437 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:56.260686 (+ 249us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:56.260752 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0929 19:00:56.260778 (+ 26us) server_negotiation.cc:299] Negotiation successful
0929 19:00:56.260840 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":85,"threads_started":1}
I20260929 19:00:56.260953 2639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:56.256499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53396 (local address 127.0.98.2:42581)
0929 19:00:56.256870 (+ 371us) server_negotiation.cc:206] Beginning negotiation
0929 19:00:56.256875 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 19:00:56.256886 (+ 11us) server_negotiation.cc:407] Connection header received
0929 19:00:56.256926 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:00:56.256927 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:00:56.256944 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:00:56.257279 (+ 335us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:00:56.257335 (+ 56us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:56.258544 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:56.259596 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:56.260297 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:56.260714 (+ 417us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:56.260782 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0929 19:00:56.260826 (+ 44us) server_negotiation.cc:299] Negotiation successful
0929 19:00:56.260858 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":72,"threads_started":1}
I20260929 19:00:56.261607 2414 tablet_service.cc:1467] Processing CreateTablet for tablet 80b7ecf16c99494e8272248d2690d89d (DEFAULT_TABLE table=test-table [id=3fe992bba5604cc7a77f241abb38a164]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:00:56.261957 2414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80b7ecf16c99494e8272248d2690d89d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:00:56.261912 2274 tablet_service.cc:1467] Processing CreateTablet for tablet 80b7ecf16c99494e8272248d2690d89d (DEFAULT_TABLE table=test-table [id=3fe992bba5604cc7a77f241abb38a164]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:00:56.262236 2274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80b7ecf16c99494e8272248d2690d89d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:00:56.265241 2644 tablet_bootstrap.cc:492] T 80b7ecf16c99494e8272248d2690d89d P 3411d1f14ac94407b9fe8e491584772b: Bootstrap starting.
I20260929 19:00:56.265372 2554 tablet_service.cc:1467] Processing CreateTablet for tablet 80b7ecf16c99494e8272248d2690d89d (DEFAULT_TABLE table=test-table [id=3fe992bba5604cc7a77f241abb38a164]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:00:56.265623 2554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80b7ecf16c99494e8272248d2690d89d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:00:56.265859 2644 tablet_bootstrap.cc:654] T 80b7ecf16c99494e8272248d2690d89d P 3411d1f14ac94407b9fe8e491584772b: Neither blocks nor log segments found. Creating new log.
I20260929 19:00:56.266149 2644 log.cc:824] T 80b7ecf16c99494e8272248d2690d89d P 3411d1f14ac94407b9fe8e491584772b: Log is configured to *not* fsync() on all Append() calls
I20260929 19:00:56.266851 2644 tablet_bootstrap.cc:492] T 80b7ecf16c99494e8272248d2690d89d P 3411d1f14ac94407b9fe8e491584772b: No bootstrap required, opened a new log
I20260929 19:00:56.266929 2644 ts_tablet_manager.cc:1397] T 80b7ecf16c99494e8272248d2690d89d P 3411d1f14ac94407b9fe8e491584772b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 19:00:56.268003 2646 tablet_bootstrap.cc:492] T 80b7ecf16c99494e8272248d2690d89d P 5fd1996f61cc4be2858c26c0f009154b: Bootstrap starting.
I20260929 19:00:56.268087 2643 tablet_bootstrap.cc:492] T 80b7ecf16c99494e8272248d2690d89d P 0725f0c1e7c74171b33e880e6a6fd5cd: Bootstrap starting.
I20260929 19:00:56.268548 2644 raft_consensus.cc:348] T 80b7ecf16c99494e8272248d2690d89d P 3411d1f14ac94407b9fe8e491584772b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3411d1f14ac94407b9fe8e491584772b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42221 } } peers { permanent_uuid: "0725f0c1e7c74171b33e880e6a6fd5cd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42581 } } peers { permanent_uuid: "5fd1996f61cc4be2858c26c0f009154b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45245 } }
I20260929 19:00:56.268698 2644 raft_consensus.cc:374] T 80b7ecf16c99494e8272248d2690d89d P 3411d1f14ac94407b9fe8e491584772b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:00:56.268718 2643 tablet_bootstrap.cc:654] T 80b7ecf16c99494e8272248d2690d89d P 0725f0c1e7c74171b33e880e6a6fd5cd: Neither blocks nor log segments found. Creating new log.
I20260929 19:00:56.268741 2644 raft_consensus.cc:729] T 80b7ecf16c99494e8272248d2690d89d P 3411d1f14ac94407b9fe8e491584772b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3411d1f14ac94407b9fe8e491584772b, State: Initialized, Role: FOLLOWER
I20260929 19:00:56.268898 2644 consensus_queue.cc:260] T 80b7ecf16c99494e8272248d2690d89d P 3411d1f14ac94407b9fe8e491584772b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3411d1f14ac94407b9fe8e491584772b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42221 } } peers { permanent_uuid: "0725f0c1e7c74171b33e880e6a6fd5cd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42581 } } peers { permanent_uuid: "5fd1996f61cc4be2858c26c0f009154b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45245 } }
I20260929 19:00:56.268930 2646 tablet_bootstrap.cc:654] T 80b7ecf16c99494e8272248d2690d89d P 5fd1996f61cc4be2858c26c0f009154b: Neither blocks nor log segments found. Creating new log.
I20260929 19:00:56.269081 2643 log.cc:824] T 80b7ecf16c99494e8272248d2690d89d P 0725f0c1e7c74171b33e880e6a6fd5cd: Log is configured to *not* fsync() on all Append() calls
I20260929 19:00:56.269164 2644 ts_tablet_manager.cc:1428] T 80b7ecf16c99494e8272248d2690d89d P 3411d1f14ac94407b9fe8e491584772b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 19:00:56.269223 2340 heartbeater.cc:503] Master 127.0.98.62:41481 was elected leader, sending a full tablet report...
I20260929 19:00:56.269258 2646 log.cc:824] T 80b7ecf16c99494e8272248d2690d89d P 5fd1996f61cc4be2858c26c0f009154b: Log is configured to *not* fsync() on all Append() calls
I20260929 19:00:56.269830 2643 tablet_bootstrap.cc:492] T 80b7ecf16c99494e8272248d2690d89d P 0725f0c1e7c74171b33e880e6a6fd5cd: No bootstrap required, opened a new log
I20260929 19:00:56.269907 2643 ts_tablet_manager.cc:1397] T 80b7ecf16c99494e8272248d2690d89d P 0725f0c1e7c74171b33e880e6a6fd5cd: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260929 19:00:56.269964 2646 tablet_bootstrap.cc:492] T 80b7ecf16c99494e8272248d2690d89d P 5fd1996f61cc4be2858c26c0f009154b: No bootstrap required, opened a new log
I20260929 19:00:56.270025 2646 ts_tablet_manager.cc:1397] T 80b7ecf16c99494e8272248d2690d89d P 5fd1996f61cc4be2858c26c0f009154b: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260929 19:00:56.271329 2643 raft_consensus.cc:348] T 80b7ecf16c99494e8272248d2690d89d P 0725f0c1e7c74171b33e880e6a6fd5cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3411d1f14ac94407b9fe8e491584772b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42221 } } peers { permanent_uuid: "0725f0c1e7c74171b33e880e6a6fd5cd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42581 } } peers { permanent_uuid: "5fd1996f61cc4be2858c26c0f009154b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45245 } }
I20260929 19:00:56.271445 2643 raft_consensus.cc:374] T 80b7ecf16c99494e8272248d2690d89d P 0725f0c1e7c74171b33e880e6a6fd5cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:00:56.271466 2643 raft_consensus.cc:729] T 80b7ecf16c99494e8272248d2690d89d P 0725f0c1e7c74171b33e880e6a6fd5cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0725f0c1e7c74171b33e880e6a6fd5cd, State: Initialized, Role: FOLLOWER
I20260929 19:00:56.271478 2646 raft_consensus.cc:348] T 80b7ecf16c99494e8272248d2690d89d P 5fd1996f61cc4be2858c26c0f009154b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3411d1f14ac94407b9fe8e491584772b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42221 } } peers { permanent_uuid: "0725f0c1e7c74171b33e880e6a6fd5cd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42581 } } peers { permanent_uuid: "5fd1996f61cc4be2858c26c0f009154b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45245 } }
I20260929 19:00:56.271570 2643 consensus_queue.cc:260] T 80b7ecf16c99494e8272248d2690d89d P 0725f0c1e7c74171b33e880e6a6fd5cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3411d1f14ac94407b9fe8e491584772b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42221 } } peers { permanent_uuid: "0725f0c1e7c74171b33e880e6a6fd5cd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42581 } } peers { permanent_uuid: "5fd1996f61cc4be2858c26c0f009154b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45245 } }
I20260929 19:00:56.271627 2646 raft_consensus.cc:374] T 80b7ecf16c99494e8272248d2690d89d P 5fd1996f61cc4be2858c26c0f009154b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:00:56.271667 2646 raft_consensus.cc:729] T 80b7ecf16c99494e8272248d2690d89d P 5fd1996f61cc4be2858c26c0f009154b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fd1996f61cc4be2858c26c0f009154b, State: Initialized, Role: FOLLOWER
I20260929 19:00:56.271781 2646 consensus_queue.cc:260] T 80b7ecf16c99494e8272248d2690d89d P 5fd1996f61cc4be2858c26c0f009154b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3411d1f14ac94407b9fe8e491584772b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42221 } } peers { permanent_uuid: "0725f0c1e7c74171b33e880e6a6fd5cd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42581 } } peers { permanent_uuid: "5fd1996f61cc4be2858c26c0f009154b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45245 } }
I20260929 19:00:56.271857 2643 ts_tablet_manager.cc:1428] T 80b7ecf16c99494e8272248d2690d89d P 0725f0c1e7c74171b33e880e6a6fd5cd: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260929 19:00:56.271860 2480 heartbeater.cc:503] Master 127.0.98.62:41481 was elected leader, sending a full tablet report...
I20260929 19:00:56.272027 2646 ts_tablet_manager.cc:1428] T 80b7ecf16c99494e8272248d2690d89d P 5fd1996f61cc4be2858c26c0f009154b: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260929 19:00:56.272059 2620 heartbeater.cc:503] Master 127.0.98.62:41481 was elected leader, sending a full tablet report...
I20260929 19:00:56.292200 2651 raft_consensus.cc:482] T 80b7ecf16c99494e8272248d2690d89d P 5fd1996f61cc4be2858c26c0f009154b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 19:00:56.292309 2651 raft_consensus.cc:504] T 80b7ecf16c99494e8272248d2690d89d P 5fd1996f61cc4be2858c26c0f009154b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3411d1f14ac94407b9fe8e491584772b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42221 } } peers { permanent_uuid: "0725f0c1e7c74171b33e880e6a6fd5cd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42581 } } peers { permanent_uuid: "5fd1996f61cc4be2858c26c0f009154b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45245 } }
I20260929 19:00:56.292636 2651 leader_election.cc:288] T 80b7ecf16c99494e8272248d2690d89d P 5fd1996f61cc4be2858c26c0f009154b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3411d1f14ac94407b9fe8e491584772b (127.0.98.1:42221), 0725f0c1e7c74171b33e880e6a6fd5cd (127.0.98.2:42581)
I20260929 19:00:56.296475 2513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:56.292851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42581 (local address 127.0.98.3:42543)
0929 19:00:56.292904 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0929 19:00:56.292916 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 19:00:56.292960 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:00:56.293117 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:00:56.293118 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:00:56.293120 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:00:56.293191 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:00:56.293193 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:00:56.294107 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:00:56.294108 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:00:56.295434 (+ 1326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:00:56.295437 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:00:56.296184 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:00:56.296185 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:00:56.296272 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:56.296276 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 19:00:56.296333 (+ 57us) client_negotiation.cc:240] Negotiation successful
0929 19:00:56.296373 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36,"mutex_wait_us":82}
I20260929 19:00:56.296500 2639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:56.292938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42543 (local address 127.0.98.2:42581)
0929 19:00:56.292985 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0929 19:00:56.292988 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 19:00:56.292998 (+ 10us) server_negotiation.cc:407] Connection header received
0929 19:00:56.293025 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:00:56.293026 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:00:56.293035 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:00:56.293094 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:00:56.293264 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:56.294031 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:56.295541 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:56.296097 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:56.296175 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:56.296186 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0929 19:00:56.296387 (+ 201us) server_negotiation.cc:299] Negotiation successful
0929 19:00:56.296433 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260929 19:00:56.296500 2641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:56.292948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35913 (local address 127.0.98.1:42221)
0929 19:00:56.292995 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0929 19:00:56.292996 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 19:00:56.293151 (+ 155us) server_negotiation.cc:407] Connection header received
0929 19:00:56.293188 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:00:56.293189 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:00:56.293198 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:00:56.293260 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:00:56.293396 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:56.294084 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:56.295540 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:56.296097 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:56.296175 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:56.296185 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0929 19:00:56.296387 (+ 202us) server_negotiation.cc:299] Negotiation successful
0929 19:00:56.296433 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260929 19:00:56.296480 2652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:56.292867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42221 (local address 127.0.98.3:35913)
0929 19:00:56.293052 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0929 19:00:56.293060 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0929 19:00:56.293138 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:00:56.293283 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:00:56.293283 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:00:56.293285 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:00:56.293333 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:00:56.293334 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:00:56.294154 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:00:56.294154 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:00:56.295434 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:00:56.295437 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:00:56.296185 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:00:56.296185 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:00:56.296272 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:56.296276 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 19:00:56.296333 (+ 57us) client_negotiation.cc:240] Negotiation successful
0929 19:00:56.296373 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"mutex_wait_us":17,"thread_start_us":100,"threads_started":1}
I20260929 19:00:56.296895 2434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80b7ecf16c99494e8272248d2690d89d" candidate_uuid: "5fd1996f61cc4be2858c26c0f009154b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0725f0c1e7c74171b33e880e6a6fd5cd" is_pre_election: true
I20260929 19:00:56.297008 2294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80b7ecf16c99494e8272248d2690d89d" candidate_uuid: "5fd1996f61cc4be2858c26c0f009154b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3411d1f14ac94407b9fe8e491584772b" is_pre_election: true
I20260929 19:00:56.297091 2434 raft_consensus.cc:2454] T 80b7ecf16c99494e8272248d2690d89d P 0725f0c1e7c74171b33e880e6a6fd5cd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5fd1996f61cc4be2858c26c0f009154b in term 0.
I20260929 19:00:56.297185 2294 raft_consensus.cc:2454] T 80b7ecf16c99494e8272248d2690d89d P 3411d1f14ac94407b9fe8e491584772b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5fd1996f61cc4be2858c26c0f009154b in term 0.
I20260929 19:00:56.297319 2505 leader_election.cc:302] T 80b7ecf16c99494e8272248d2690d89d P 5fd1996f61cc4be2858c26c0f009154b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0725f0c1e7c74171b33e880e6a6fd5cd, 5fd1996f61cc4be2858c26c0f009154b; no voters:
I20260929 19:00:56.297448 2651 raft_consensus.cc:2781] T 80b7ecf16c99494e8272248d2690d89d P 5fd1996f61cc4be2858c26c0f009154b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 19:00:56.297519 2651 raft_consensus.cc:482] T 80b7ecf16c99494e8272248d2690d89d P 5fd1996f61cc4be2858c26c0f009154b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 19:00:56.297537 2651 raft_consensus.cc:3037] T 80b7ecf16c99494e8272248d2690d89d P 5fd1996f61cc4be2858c26c0f009154b [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:00:56.298243 2651 raft_consensus.cc:504] T 80b7ecf16c99494e8272248d2690d89d P 5fd1996f61cc4be2858c26c0f009154b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3411d1f14ac94407b9fe8e491584772b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42221 } } peers { permanent_uuid: "0725f0c1e7c74171b33e880e6a6fd5cd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42581 } } peers { permanent_uuid: "5fd1996f61cc4be2858c26c0f009154b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45245 } }
I20260929 19:00:56.298395 2651 leader_election.cc:288] T 80b7ecf16c99494e8272248d2690d89d P 5fd1996f61cc4be2858c26c0f009154b [CANDIDATE]: Term 1 election: Requested vote from peers 3411d1f14ac94407b9fe8e491584772b (127.0.98.1:42221), 0725f0c1e7c74171b33e880e6a6fd5cd (127.0.98.2:42581)
I20260929 19:00:56.298568 2294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80b7ecf16c99494e8272248d2690d89d" candidate_uuid: "5fd1996f61cc4be2858c26c0f009154b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3411d1f14ac94407b9fe8e491584772b"
I20260929 19:00:56.298568 2434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80b7ecf16c99494e8272248d2690d89d" candidate_uuid: "5fd1996f61cc4be2858c26c0f009154b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0725f0c1e7c74171b33e880e6a6fd5cd"
I20260929 19:00:56.298645 2434 raft_consensus.cc:3037] T 80b7ecf16c99494e8272248d2690d89d P 0725f0c1e7c74171b33e880e6a6fd5cd [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:00:56.298646 2294 raft_consensus.cc:3037] T 80b7ecf16c99494e8272248d2690d89d P 3411d1f14ac94407b9fe8e491584772b [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:00:56.299489 2294 raft_consensus.cc:2454] T 80b7ecf16c99494e8272248d2690d89d P 3411d1f14ac94407b9fe8e491584772b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5fd1996f61cc4be2858c26c0f009154b in term 1.
I20260929 19:00:56.299487 2434 raft_consensus.cc:2454] T 80b7ecf16c99494e8272248d2690d89d P 0725f0c1e7c74171b33e880e6a6fd5cd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5fd1996f61cc4be2858c26c0f009154b in term 1.
I20260929 19:00:56.299660 2506 leader_election.cc:302] T 80b7ecf16c99494e8272248d2690d89d P 5fd1996f61cc4be2858c26c0f009154b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3411d1f14ac94407b9fe8e491584772b, 5fd1996f61cc4be2858c26c0f009154b; no voters:
I20260929 19:00:56.299762 2651 raft_consensus.cc:2781] T 80b7ecf16c99494e8272248d2690d89d P 5fd1996f61cc4be2858c26c0f009154b [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:00:56.299903 2651 raft_consensus.cc:686] T 80b7ecf16c99494e8272248d2690d89d P 5fd1996f61cc4be2858c26c0f009154b [term 1 LEADER]: Becoming Leader. State: Replica: 5fd1996f61cc4be2858c26c0f009154b, State: Running, Role: LEADER
I20260929 19:00:56.300024 2651 consensus_queue.cc:237] T 80b7ecf16c99494e8272248d2690d89d P 5fd1996f61cc4be2858c26c0f009154b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3411d1f14ac94407b9fe8e491584772b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42221 } } peers { permanent_uuid: "0725f0c1e7c74171b33e880e6a6fd5cd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42581 } } peers { permanent_uuid: "5fd1996f61cc4be2858c26c0f009154b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45245 } }
I20260929 19:00:56.300945 2136 catalog_manager.cc:5495] T 80b7ecf16c99494e8272248d2690d89d P 5fd1996f61cc4be2858c26c0f009154b reported cstate change: term changed from 0 to 1, leader changed from <none> to 5fd1996f61cc4be2858c26c0f009154b (127.0.98.3). New cstate: current_term: 1 leader_uuid: "5fd1996f61cc4be2858c26c0f009154b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3411d1f14ac94407b9fe8e491584772b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0725f0c1e7c74171b33e880e6a6fd5cd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42581 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5fd1996f61cc4be2858c26c0f009154b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45245 } health_report { overall_health: HEALTHY } } }
W20260929 19:00:56.311538 2481 tablet.cc:2369] T 80b7ecf16c99494e8272248d2690d89d P 0725f0c1e7c74171b33e880e6a6fd5cd: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 29 19:00:56 dist-test-slave-lqkv krb5kdc[2092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708455, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 19:00:56.347853 2187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:56.341588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42840 (local address 127.0.98.62:41481)
0929 19:00:56.341660 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0929 19:00:56.341662 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:00:56.341745 (+ 83us) server_negotiation.cc:407] Connection header received
0929 19:00:56.341797 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:00:56.341798 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:00:56.341806 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:00:56.341856 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:00:56.342270 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:56.342733 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:56.343469 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:56.343650 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:56.343703 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:56.345773 (+ 2070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:00:56.345774 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:00:56.345774 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:00:56.345780 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:00:56.346909 (+ 1129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:00:56.347235 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:00:56.347235 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:00:56.347236 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:00:56.347284 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:00:56.347448 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:00:56.347449 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:00:56.347449 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:00:56.347542 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:00:56.347589 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0929 19:00:56.347717 (+ 128us) server_negotiation.cc:299] Negotiation successful
0929 19:00:56.347792 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 19:00:56.348001 2660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:56.341504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41481 (local address 127.0.0.1:42840)
0929 19:00:56.341655 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0929 19:00:56.341665 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 19:00:56.341740 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:00:56.341873 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:00:56.341874 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:00:56.342163 (+ 289us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:00:56.342208 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:00:56.342210 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:00:56.342796 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:00:56.342797 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:00:56.343384 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:00:56.343386 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:00:56.343712 (+ 326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:00:56.343712 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:00:56.343766 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:56.344219 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:00:56.344223 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:00:56.345678 (+ 1455us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:00:56.346976 (+ 1298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:00:56.346982 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:00:56.346984 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:00:56.347171 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:00:56.347341 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:00:56.347342 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:00:56.347342 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:00:56.347396 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:00:56.347606 (+ 210us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:00:56.347611 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:00:56.347650 (+ 39us) client_negotiation.cc:769] Sending connection context
0929 19:00:56.347702 (+ 52us) client_negotiation.cc:240] Negotiation successful
0929 19:00:56.347859 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"mutex_wait_us":78,"thread_start_us":77,"threads_started":1}
I20260929 19:00:56.349782 2136 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42840:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260929 19:00:56.349975 2136 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 19:00:56.351374 2274 tablet_service.cc:1467] Processing CreateTablet for tablet 9b4493583a9f41f6a79cca136c2a924b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a4a74c8df4d54429926c9668d300394e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 19:00:56.351553 2274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b4493583a9f41f6a79cca136c2a924b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:00:56.351557 2554 tablet_service.cc:1467] Processing CreateTablet for tablet 9b4493583a9f41f6a79cca136c2a924b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a4a74c8df4d54429926c9668d300394e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 19:00:56.351712 2554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b4493583a9f41f6a79cca136c2a924b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:00:56.352823 2644 tablet_bootstrap.cc:492] T 9b4493583a9f41f6a79cca136c2a924b P 3411d1f14ac94407b9fe8e491584772b: Bootstrap starting.
I20260929 19:00:56.352963 2414 tablet_service.cc:1467] Processing CreateTablet for tablet 9b4493583a9f41f6a79cca136c2a924b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a4a74c8df4d54429926c9668d300394e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 19:00:56.353037 2646 tablet_bootstrap.cc:492] T 9b4493583a9f41f6a79cca136c2a924b P 5fd1996f61cc4be2858c26c0f009154b: Bootstrap starting.
I20260929 19:00:56.353123 2414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b4493583a9f41f6a79cca136c2a924b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:00:56.353497 2646 tablet_bootstrap.cc:654] T 9b4493583a9f41f6a79cca136c2a924b P 5fd1996f61cc4be2858c26c0f009154b: Neither blocks nor log segments found. Creating new log.
I20260929 19:00:56.353506 2644 tablet_bootstrap.cc:654] T 9b4493583a9f41f6a79cca136c2a924b P 3411d1f14ac94407b9fe8e491584772b: Neither blocks nor log segments found. Creating new log.
I20260929 19:00:56.354200 2646 tablet_bootstrap.cc:492] T 9b4493583a9f41f6a79cca136c2a924b P 5fd1996f61cc4be2858c26c0f009154b: No bootstrap required, opened a new log
I20260929 19:00:56.354218 2644 tablet_bootstrap.cc:492] T 9b4493583a9f41f6a79cca136c2a924b P 3411d1f14ac94407b9fe8e491584772b: No bootstrap required, opened a new log
I20260929 19:00:56.354262 2646 ts_tablet_manager.cc:1397] T 9b4493583a9f41f6a79cca136c2a924b P 5fd1996f61cc4be2858c26c0f009154b: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260929 19:00:56.354279 2644 ts_tablet_manager.cc:1397] T 9b4493583a9f41f6a79cca136c2a924b P 3411d1f14ac94407b9fe8e491584772b: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260929 19:00:56.354429 2646 raft_consensus.cc:348] T 9b4493583a9f41f6a79cca136c2a924b P 5fd1996f61cc4be2858c26c0f009154b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3411d1f14ac94407b9fe8e491584772b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42221 } } peers { permanent_uuid: "0725f0c1e7c74171b33e880e6a6fd5cd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42581 } } peers { permanent_uuid: "5fd1996f61cc4be2858c26c0f009154b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45245 } }
I20260929 19:00:56.354424 2644 raft_consensus.cc:348] T 9b4493583a9f41f6a79cca136c2a924b P 3411d1f14ac94407b9fe8e491584772b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3411d1f14ac94407b9fe8e491584772b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42221 } } peers { permanent_uuid: "0725f0c1e7c74171b33e880e6a6fd5cd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42581 } } peers { permanent_uuid: "5fd1996f61cc4be2858c26c0f009154b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45245 } }
I20260929 19:00:56.354501 2643 tablet_bootstrap.cc:492] T 9b4493583a9f41f6a79cca136c2a924b P 0725f0c1e7c74171b33e880e6a6fd5cd: Bootstrap starting.
I20260929 19:00:56.354508 2646 raft_consensus.cc:374] T 9b4493583a9f41f6a79cca136c2a924b P 5fd1996f61cc4be2858c26c0f009154b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:00:56.354528 2644 raft_consensus.cc:374] T 9b4493583a9f41f6a79cca136c2a924b P 3411d1f14ac94407b9fe8e491584772b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:00:56.354537 2646 raft_consensus.cc:729] T 9b4493583a9f41f6a79cca136c2a924b P 5fd1996f61cc4be2858c26c0f009154b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fd1996f61cc4be2858c26c0f009154b, State: Initialized, Role: FOLLOWER
I20260929 19:00:56.354561 2644 raft_consensus.cc:729] T 9b4493583a9f41f6a79cca136c2a924b P 3411d1f14ac94407b9fe8e491584772b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3411d1f14ac94407b9fe8e491584772b, State: Initialized, Role: FOLLOWER
I20260929 19:00:56.354583 2646 consensus_queue.cc:260] T 9b4493583a9f41f6a79cca136c2a924b P 5fd1996f61cc4be2858c26c0f009154b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3411d1f14ac94407b9fe8e491584772b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42221 } } peers { permanent_uuid: "0725f0c1e7c74171b33e880e6a6fd5cd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42581 } } peers { permanent_uuid: "5fd1996f61cc4be2858c26c0f009154b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45245 } }
I20260929 19:00:56.354604 2644 consensus_queue.cc:260] T 9b4493583a9f41f6a79cca136c2a924b P 3411d1f14ac94407b9fe8e491584772b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3411d1f14ac94407b9fe8e491584772b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42221 } } peers { permanent_uuid: "0725f0c1e7c74171b33e880e6a6fd5cd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42581 } } peers { permanent_uuid: "5fd1996f61cc4be2858c26c0f009154b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45245 } }
I20260929 19:00:56.354681 2646 ts_tablet_manager.cc:1428] T 9b4493583a9f41f6a79cca136c2a924b P 5fd1996f61cc4be2858c26c0f009154b: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260929 19:00:56.354713 2644 ts_tablet_manager.cc:1428] T 9b4493583a9f41f6a79cca136c2a924b P 3411d1f14ac94407b9fe8e491584772b: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260929 19:00:56.354887 2647 tablet_replica.cc:439] T 9b4493583a9f41f6a79cca136c2a924b P 3411d1f14ac94407b9fe8e491584772b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3411d1f14ac94407b9fe8e491584772b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42221 } } peers { permanent_uuid: "0725f0c1e7c74171b33e880e6a6fd5cd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42581 } } peers { permanent_uuid: "5fd1996f61cc4be2858c26c0f009154b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45245 } } }
I20260929 19:00:56.354971 2647 tablet_replica.cc:442] T 9b4493583a9f41f6a79cca136c2a924b P 3411d1f14ac94407b9fe8e491584772b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 19:00:56.355175 2643 tablet_bootstrap.cc:654] T 9b4493583a9f41f6a79cca136c2a924b P 0725f0c1e7c74171b33e880e6a6fd5cd: Neither blocks nor log segments found. Creating new log.
I20260929 19:00:56.355510 2651 tablet_replica.cc:439] T 9b4493583a9f41f6a79cca136c2a924b P 5fd1996f61cc4be2858c26c0f009154b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3411d1f14ac94407b9fe8e491584772b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42221 } } peers { permanent_uuid: "0725f0c1e7c74171b33e880e6a6fd5cd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42581 } } peers { permanent_uuid: "5fd1996f61cc4be2858c26c0f009154b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45245 } } }
I20260929 19:00:56.355612 2651 tablet_replica.cc:442] T 9b4493583a9f41f6a79cca136c2a924b P 5fd1996f61cc4be2858c26c0f009154b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 19:00:56.355899 2643 tablet_bootstrap.cc:492] T 9b4493583a9f41f6a79cca136c2a924b P 0725f0c1e7c74171b33e880e6a6fd5cd: No bootstrap required, opened a new log
I20260929 19:00:56.355949 2643 ts_tablet_manager.cc:1397] T 9b4493583a9f41f6a79cca136c2a924b P 0725f0c1e7c74171b33e880e6a6fd5cd: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260929 19:00:56.356096 2643 raft_consensus.cc:348] T 9b4493583a9f41f6a79cca136c2a924b P 0725f0c1e7c74171b33e880e6a6fd5cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3411d1f14ac94407b9fe8e491584772b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42221 } } peers { permanent_uuid: "0725f0c1e7c74171b33e880e6a6fd5cd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42581 } } peers { permanent_uuid: "5fd1996f61cc4be2858c26c0f009154b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45245 } }
I20260929 19:00:56.356164 2643 raft_consensus.cc:374] T 9b4493583a9f41f6a79cca136c2a924b P 0725f0c1e7c74171b33e880e6a6fd5cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:00:56.356184 2643 raft_consensus.cc:729] T 9b4493583a9f41f6a79cca136c2a924b P 0725f0c1e7c74171b33e880e6a6fd5cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0725f0c1e7c74171b33e880e6a6fd5cd, State: Initialized, Role: FOLLOWER
I20260929 19:00:56.356225 2643 consensus_queue.cc:260] T 9b4493583a9f41f6a79cca136c2a924b P 0725f0c1e7c74171b33e880e6a6fd5cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3411d1f14ac94407b9fe8e491584772b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42221 } } peers { permanent_uuid: "0725f0c1e7c74171b33e880e6a6fd5cd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42581 } } peers { permanent_uuid: "5fd1996f61cc4be2858c26c0f009154b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45245 } }
I20260929 19:00:56.356328 2643 ts_tablet_manager.cc:1428] T 9b4493583a9f41f6a79cca136c2a924b P 0725f0c1e7c74171b33e880e6a6fd5cd: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260929 19:00:56.356397 2650 tablet_replica.cc:439] T 9b4493583a9f41f6a79cca136c2a924b P 0725f0c1e7c74171b33e880e6a6fd5cd: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3411d1f14ac94407b9fe8e491584772b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42221 } } peers { permanent_uuid: "0725f0c1e7c74171b33e880e6a6fd5cd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42581 } } peers { permanent_uuid: "5fd1996f61cc4be2858c26c0f009154b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45245 } } }
I20260929 19:00:56.356490 2650 tablet_replica.cc:442] T 9b4493583a9f41f6a79cca136c2a924b P 0725f0c1e7c74171b33e880e6a6fd5cd: This TxnStatusTablet replica's current role is: FOLLOWER
W20260929 19:00:56.411440 2341 tablet.cc:2369] T 80b7ecf16c99494e8272248d2690d89d P 3411d1f14ac94407b9fe8e491584772b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 19:00:56.470973 2621 tablet.cc:2369] T 80b7ecf16c99494e8272248d2690d89d P 5fd1996f61cc4be2858c26c0f009154b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 19:00:56.579528 2651 raft_consensus.cc:482] T 9b4493583a9f41f6a79cca136c2a924b P 5fd1996f61cc4be2858c26c0f009154b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 19:00:56.579610 2651 raft_consensus.cc:504] T 9b4493583a9f41f6a79cca136c2a924b P 5fd1996f61cc4be2858c26c0f009154b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3411d1f14ac94407b9fe8e491584772b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42221 } } peers { permanent_uuid: "0725f0c1e7c74171b33e880e6a6fd5cd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42581 } } peers { permanent_uuid: "5fd1996f61cc4be2858c26c0f009154b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45245 } }
I20260929 19:00:56.579770 2651 leader_election.cc:288] T 9b4493583a9f41f6a79cca136c2a924b P 5fd1996f61cc4be2858c26c0f009154b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3411d1f14ac94407b9fe8e491584772b (127.0.98.1:42221), 0725f0c1e7c74171b33e880e6a6fd5cd (127.0.98.2:42581)
I20260929 19:00:56.579943 2434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b4493583a9f41f6a79cca136c2a924b" candidate_uuid: "5fd1996f61cc4be2858c26c0f009154b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0725f0c1e7c74171b33e880e6a6fd5cd" is_pre_election: true
I20260929 19:00:56.579918 2294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b4493583a9f41f6a79cca136c2a924b" candidate_uuid: "5fd1996f61cc4be2858c26c0f009154b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3411d1f14ac94407b9fe8e491584772b" is_pre_election: true
I20260929 19:00:56.580029 2434 raft_consensus.cc:2454] T 9b4493583a9f41f6a79cca136c2a924b P 0725f0c1e7c74171b33e880e6a6fd5cd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5fd1996f61cc4be2858c26c0f009154b in term 0.
I20260929 19:00:56.580029 2294 raft_consensus.cc:2454] T 9b4493583a9f41f6a79cca136c2a924b P 3411d1f14ac94407b9fe8e491584772b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5fd1996f61cc4be2858c26c0f009154b in term 0.
I20260929 19:00:56.580159 2505 leader_election.cc:302] T 9b4493583a9f41f6a79cca136c2a924b P 5fd1996f61cc4be2858c26c0f009154b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0725f0c1e7c74171b33e880e6a6fd5cd, 5fd1996f61cc4be2858c26c0f009154b; no voters:
I20260929 19:00:56.580283 2651 raft_consensus.cc:2781] T 9b4493583a9f41f6a79cca136c2a924b P 5fd1996f61cc4be2858c26c0f009154b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 19:00:56.580334 2651 raft_consensus.cc:482] T 9b4493583a9f41f6a79cca136c2a924b P 5fd1996f61cc4be2858c26c0f009154b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 19:00:56.580365 2651 raft_consensus.cc:3037] T 9b4493583a9f41f6a79cca136c2a924b P 5fd1996f61cc4be2858c26c0f009154b [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:00:56.581197 2651 raft_consensus.cc:504] T 9b4493583a9f41f6a79cca136c2a924b P 5fd1996f61cc4be2858c26c0f009154b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3411d1f14ac94407b9fe8e491584772b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42221 } } peers { permanent_uuid: "0725f0c1e7c74171b33e880e6a6fd5cd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42581 } } peers { permanent_uuid: "5fd1996f61cc4be2858c26c0f009154b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45245 } }
I20260929 19:00:56.581342 2651 leader_election.cc:288] T 9b4493583a9f41f6a79cca136c2a924b P 5fd1996f61cc4be2858c26c0f009154b [CANDIDATE]: Term 1 election: Requested vote from peers 3411d1f14ac94407b9fe8e491584772b (127.0.98.1:42221), 0725f0c1e7c74171b33e880e6a6fd5cd (127.0.98.2:42581)
I20260929 19:00:56.581489 2294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b4493583a9f41f6a79cca136c2a924b" candidate_uuid: "5fd1996f61cc4be2858c26c0f009154b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3411d1f14ac94407b9fe8e491584772b"
I20260929 19:00:56.581561 2294 raft_consensus.cc:3037] T 9b4493583a9f41f6a79cca136c2a924b P 3411d1f14ac94407b9fe8e491584772b [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:00:56.581512 2434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b4493583a9f41f6a79cca136c2a924b" candidate_uuid: "5fd1996f61cc4be2858c26c0f009154b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0725f0c1e7c74171b33e880e6a6fd5cd"
I20260929 19:00:56.581604 2434 raft_consensus.cc:3037] T 9b4493583a9f41f6a79cca136c2a924b P 0725f0c1e7c74171b33e880e6a6fd5cd [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:00:56.582262 2294 raft_consensus.cc:2454] T 9b4493583a9f41f6a79cca136c2a924b P 3411d1f14ac94407b9fe8e491584772b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5fd1996f61cc4be2858c26c0f009154b in term 1.
I20260929 19:00:56.582335 2434 raft_consensus.cc:2454] T 9b4493583a9f41f6a79cca136c2a924b P 0725f0c1e7c74171b33e880e6a6fd5cd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5fd1996f61cc4be2858c26c0f009154b in term 1.
I20260929 19:00:56.582436 2506 leader_election.cc:302] T 9b4493583a9f41f6a79cca136c2a924b P 5fd1996f61cc4be2858c26c0f009154b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3411d1f14ac94407b9fe8e491584772b, 5fd1996f61cc4be2858c26c0f009154b; no voters:
I20260929 19:00:56.582544 2651 raft_consensus.cc:2781] T 9b4493583a9f41f6a79cca136c2a924b P 5fd1996f61cc4be2858c26c0f009154b [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:00:56.582594 2651 raft_consensus.cc:686] T 9b4493583a9f41f6a79cca136c2a924b P 5fd1996f61cc4be2858c26c0f009154b [term 1 LEADER]: Becoming Leader. State: Replica: 5fd1996f61cc4be2858c26c0f009154b, State: Running, Role: LEADER
I20260929 19:00:56.582686 2651 consensus_queue.cc:237] T 9b4493583a9f41f6a79cca136c2a924b P 5fd1996f61cc4be2858c26c0f009154b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3411d1f14ac94407b9fe8e491584772b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42221 } } peers { permanent_uuid: "0725f0c1e7c74171b33e880e6a6fd5cd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42581 } } peers { permanent_uuid: "5fd1996f61cc4be2858c26c0f009154b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45245 } }
I20260929 19:00:56.583031 2653 tablet_replica.cc:439] T 9b4493583a9f41f6a79cca136c2a924b P 5fd1996f61cc4be2858c26c0f009154b: TxnStatusTablet state changed. Reason: New leader 5fd1996f61cc4be2858c26c0f009154b. Latest consensus state: current_term: 1 leader_uuid: "5fd1996f61cc4be2858c26c0f009154b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3411d1f14ac94407b9fe8e491584772b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42221 } } peers { permanent_uuid: "0725f0c1e7c74171b33e880e6a6fd5cd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42581 } } peers { permanent_uuid: "5fd1996f61cc4be2858c26c0f009154b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45245 } } }
I20260929 19:00:56.583139 2653 tablet_replica.cc:442] T 9b4493583a9f41f6a79cca136c2a924b P 5fd1996f61cc4be2858c26c0f009154b: This TxnStatusTablet replica's current role is: LEADER
I20260929 19:00:56.583242 2137 catalog_manager.cc:5495] T 9b4493583a9f41f6a79cca136c2a924b P 5fd1996f61cc4be2858c26c0f009154b reported cstate change: term changed from 0 to 1, leader changed from <none> to 5fd1996f61cc4be2858c26c0f009154b (127.0.98.3). New cstate: current_term: 1 leader_uuid: "5fd1996f61cc4be2858c26c0f009154b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3411d1f14ac94407b9fe8e491584772b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0725f0c1e7c74171b33e880e6a6fd5cd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42581 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5fd1996f61cc4be2858c26c0f009154b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45245 } health_report { overall_health: HEALTHY } } }
I20260929 19:00:56.583359 2665 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260929 19:00:56.583477 2665 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260929 19:00:56.639142 2642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:56.636723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49882 (local address 127.0.98.3:45245)
0929 19:00:56.636768 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0929 19:00:56.636770 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:00:56.636779 (+ 9us) server_negotiation.cc:407] Connection header received
0929 19:00:56.636897 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:00:56.636898 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:00:56.636916 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:00:56.636976 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 19:00:56.637121 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:56.637588 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:56.638344 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:56.638550 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:56.638625 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:56.638808 (+ 183us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 19:00:56.638903 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 19:00:56.638993 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0929 19:00:56.639034 (+ 41us) server_negotiation.cc:299] Negotiation successful
0929 19:00:56.639054 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260929 19:00:56.639150 2660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:56.636669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45245 (local address 127.0.0.1:49882)
0929 19:00:56.636732 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0929 19:00:56.636745 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 19:00:56.636795 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:00:56.636994 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:00:56.636996 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:00:56.636997 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 19:00:56.637046 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:00:56.637048 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:00:56.637691 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:00:56.637692 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:00:56.638263 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:00:56.638265 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:00:56.638623 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:00:56.638624 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:00:56.638687 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:56.638704 (+ 17us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 19:00:56.638970 (+ 266us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 19:00:56.638979 (+ 9us) client_negotiation.cc:769] Sending connection context
0929 19:00:56.639028 (+ 49us) client_negotiation.cc:240] Negotiation successful
0929 19:00:56.639073 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24}
I20260929 19:00:56.640959 2434 raft_consensus.cc:1261] T 9b4493583a9f41f6a79cca136c2a924b P 0725f0c1e7c74171b33e880e6a6fd5cd [term 1 FOLLOWER]: Refusing update from remote peer 5fd1996f61cc4be2858c26c0f009154b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 19:00:56.641009 2294 raft_consensus.cc:1261] T 9b4493583a9f41f6a79cca136c2a924b P 3411d1f14ac94407b9fe8e491584772b [term 1 FOLLOWER]: Refusing update from remote peer 5fd1996f61cc4be2858c26c0f009154b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 19:00:56.641342 2651 consensus_queue.cc:1035] T 9b4493583a9f41f6a79cca136c2a924b P 5fd1996f61cc4be2858c26c0f009154b [LEADER]: Connected to new peer: Peer: permanent_uuid: "3411d1f14ac94407b9fe8e491584772b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 19:00:56.641472 2653 consensus_queue.cc:1035] T 9b4493583a9f41f6a79cca136c2a924b P 5fd1996f61cc4be2858c26c0f009154b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0725f0c1e7c74171b33e880e6a6fd5cd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 19:00:56.642519 2647 tablet_replica.cc:439] T 9b4493583a9f41f6a79cca136c2a924b P 3411d1f14ac94407b9fe8e491584772b: TxnStatusTablet state changed. Reason: New leader 5fd1996f61cc4be2858c26c0f009154b. Latest consensus state: current_term: 1 leader_uuid: "5fd1996f61cc4be2858c26c0f009154b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3411d1f14ac94407b9fe8e491584772b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42221 } } peers { permanent_uuid: "0725f0c1e7c74171b33e880e6a6fd5cd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42581 } } peers { permanent_uuid: "5fd1996f61cc4be2858c26c0f009154b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45245 } } }
I20260929 19:00:56.642551 2650 tablet_replica.cc:439] T 9b4493583a9f41f6a79cca136c2a924b P 0725f0c1e7c74171b33e880e6a6fd5cd: TxnStatusTablet state changed. Reason: New leader 5fd1996f61cc4be2858c26c0f009154b. Latest consensus state: current_term: 1 leader_uuid: "5fd1996f61cc4be2858c26c0f009154b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3411d1f14ac94407b9fe8e491584772b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42221 } } peers { permanent_uuid: "0725f0c1e7c74171b33e880e6a6fd5cd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42581 } } peers { permanent_uuid: "5fd1996f61cc4be2858c26c0f009154b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45245 } } }
I20260929 19:00:56.642613 2647 tablet_replica.cc:442] T 9b4493583a9f41f6a79cca136c2a924b P 3411d1f14ac94407b9fe8e491584772b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 19:00:56.642630 2650 tablet_replica.cc:442] T 9b4493583a9f41f6a79cca136c2a924b P 0725f0c1e7c74171b33e880e6a6fd5cd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 19:00:56.643705 2653 tablet_replica.cc:439] T 9b4493583a9f41f6a79cca136c2a924b P 5fd1996f61cc4be2858c26c0f009154b: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5fd1996f61cc4be2858c26c0f009154b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3411d1f14ac94407b9fe8e491584772b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42221 } } peers { permanent_uuid: "0725f0c1e7c74171b33e880e6a6fd5cd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42581 } } peers { permanent_uuid: "5fd1996f61cc4be2858c26c0f009154b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45245 } } }
I20260929 19:00:56.643784 2653 tablet_replica.cc:442] T 9b4493583a9f41f6a79cca136c2a924b P 5fd1996f61cc4be2858c26c0f009154b: This TxnStatusTablet replica's current role is: LEADER
I20260929 19:00:56.643838 2653 tablet_replica.cc:439] T 9b4493583a9f41f6a79cca136c2a924b P 5fd1996f61cc4be2858c26c0f009154b: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5fd1996f61cc4be2858c26c0f009154b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3411d1f14ac94407b9fe8e491584772b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42221 } } peers { permanent_uuid: "0725f0c1e7c74171b33e880e6a6fd5cd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42581 } } peers { permanent_uuid: "5fd1996f61cc4be2858c26c0f009154b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45245 } } }
I20260929 19:00:56.643889 2653 tablet_replica.cc:442] T 9b4493583a9f41f6a79cca136c2a924b P 5fd1996f61cc4be2858c26c0f009154b: This TxnStatusTablet replica's current role is: LEADER
I20260929 19:00:56.644034 2669 mvcc.cc:205] Tried to move back new op lower bound from 7334741838398623744 to 7334741838163070976. Current Snapshot: MvccSnapshot[applied={T|T < 7334741838398623744}]
I20260929 19:00:56.644140 2647 tablet_replica.cc:439] T 9b4493583a9f41f6a79cca136c2a924b P 3411d1f14ac94407b9fe8e491584772b: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5fd1996f61cc4be2858c26c0f009154b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3411d1f14ac94407b9fe8e491584772b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42221 } } peers { permanent_uuid: "0725f0c1e7c74171b33e880e6a6fd5cd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42581 } } peers { permanent_uuid: "5fd1996f61cc4be2858c26c0f009154b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45245 } } }
I20260929 19:00:56.644205 2647 tablet_replica.cc:442] T 9b4493583a9f41f6a79cca136c2a924b P 3411d1f14ac94407b9fe8e491584772b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 19:00:56.645092 2650 tablet_replica.cc:439] T 9b4493583a9f41f6a79cca136c2a924b P 0725f0c1e7c74171b33e880e6a6fd5cd: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5fd1996f61cc4be2858c26c0f009154b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3411d1f14ac94407b9fe8e491584772b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42221 } } peers { permanent_uuid: "0725f0c1e7c74171b33e880e6a6fd5cd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42581 } } peers { permanent_uuid: "5fd1996f61cc4be2858c26c0f009154b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45245 } } }
I20260929 19:00:56.645203 2650 tablet_replica.cc:442] T 9b4493583a9f41f6a79cca136c2a924b P 0725f0c1e7c74171b33e880e6a6fd5cd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 19:00:56.645404 2670 mvcc.cc:205] Tried to move back new op lower bound from 7334741838398623744 to 7334741838163070976. Current Snapshot: MvccSnapshot[applied={T|T < 7334741838398623744}]
I20260929 19:00:56.647987 2642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:56.645625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49894 (local address 127.0.98.3:45245)
0929 19:00:56.645669 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0929 19:00:56.645671 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:00:56.645680 (+ 9us) server_negotiation.cc:407] Connection header received
0929 19:00:56.645749 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:00:56.645750 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:00:56.645757 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:00:56.645798 (+ 41us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 19:00:56.645911 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:56.646334 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:56.647407 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:56.647585 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:56.647630 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:56.647798 (+ 168us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 19:00:56.647820 (+ 22us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 19:00:56.647853 (+ 33us) server_negotiation.cc:1035] Waiting for connection context
0929 19:00:56.647931 (+ 78us) server_negotiation.cc:299] Negotiation successful
0929 19:00:56.647944 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260929 19:00:56.652480 2642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:56.650343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41303 (local address 127.0.98.3:45245)
0929 19:00:56.650381 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0929 19:00:56.650382 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 19:00:56.650390 (+ 8us) server_negotiation.cc:407] Connection header received
0929 19:00:56.650450 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:00:56.650451 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:00:56.650459 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:00:56.650509 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 19:00:56.650633 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:56.651049 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:56.651785 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:56.651966 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:56.652043 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:56.652218 (+ 175us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 19:00:56.652247 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 19:00:56.652308 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0929 19:00:56.652357 (+ 49us) server_negotiation.cc:299] Negotiation successful
0929 19:00:56.652386 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":23}
I20260929 19:00:56.652480 2626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:56.650282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45245 (local address 127.0.98.3:41303)
0929 19:00:56.650350 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0929 19:00:56.650361 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 19:00:56.650402 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:00:56.650512 (+ 110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:00:56.650513 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:00:56.650514 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 19:00:56.650566 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:00:56.650568 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:00:56.651124 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:00:56.651125 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:00:56.651685 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:00:56.651687 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:00:56.652043 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:00:56.652044 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:00:56.652101 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:56.652113 (+ 12us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 19:00:56.652307 (+ 194us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 19:00:56.652308 (+ 1us) client_negotiation.cc:769] Sending connection context
0929 19:00:56.652356 (+ 48us) client_negotiation.cc:240] Negotiation successful
0929 19:00:56.652403 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36,"mutex_wait_us":4}
I20260929 19:00:56.654630 2434 raft_consensus.cc:1261] T 80b7ecf16c99494e8272248d2690d89d P 0725f0c1e7c74171b33e880e6a6fd5cd [term 1 FOLLOWER]: Refusing update from remote peer 5fd1996f61cc4be2858c26c0f009154b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 19:00:56.654685 2294 raft_consensus.cc:1261] T 80b7ecf16c99494e8272248d2690d89d P 3411d1f14ac94407b9fe8e491584772b [term 1 FOLLOWER]: Refusing update from remote peer 5fd1996f61cc4be2858c26c0f009154b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 19:00:56.654882 2653 consensus_queue.cc:1035] T 80b7ecf16c99494e8272248d2690d89d P 5fd1996f61cc4be2858c26c0f009154b [LEADER]: Connected to new peer: Peer: permanent_uuid: "3411d1f14ac94407b9fe8e491584772b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 19:00:56.654973 2651 consensus_queue.cc:1035] T 80b7ecf16c99494e8272248d2690d89d P 5fd1996f61cc4be2858c26c0f009154b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0725f0c1e7c74171b33e880e6a6fd5cd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 19:00:56.656363 2668 mvcc.cc:205] Tried to move back new op lower bound from 7334741838456328192 to 7334741837005627392. Current Snapshot: MvccSnapshot[applied={T|T < 7334741838456328192}]
I20260929 19:00:56.673023 2136 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42830:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 19:00:56.673135 2136 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42830:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 19:00:56.673898 2136 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 624b000e6c514bc39d63e63ee983eca5: Sending DeleteTablet for 3 replicas of tablet 80b7ecf16c99494e8272248d2690d89d
I20260929 19:00:56.674211 2414 tablet_service.cc:1514] Processing DeleteTablet for tablet 80b7ecf16c99494e8272248d2690d89d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:00:56 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53396
I20260929 19:00:56.674213 2274 tablet_service.cc:1514] Processing DeleteTablet for tablet 80b7ecf16c99494e8272248d2690d89d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:00:56 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55212
I20260929 19:00:56.674465 2684 tablet_replica.cc:330] T 80b7ecf16c99494e8272248d2690d89d P 3411d1f14ac94407b9fe8e491584772b: stopping tablet replica
I20260929 19:00:56.674449 2553 tablet_service.cc:1514] Processing DeleteTablet for tablet 80b7ecf16c99494e8272248d2690d89d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:00:56 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49880
I20260929 19:00:56.674571 2684 raft_consensus.cc:2229] T 80b7ecf16c99494e8272248d2690d89d P 3411d1f14ac94407b9fe8e491584772b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 19:00:56.674643 2684 raft_consensus.cc:2258] T 80b7ecf16c99494e8272248d2690d89d P 3411d1f14ac94407b9fe8e491584772b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 19:00:56.674751 2685 tablet_replica.cc:330] T 80b7ecf16c99494e8272248d2690d89d P 5fd1996f61cc4be2858c26c0f009154b: stopping tablet replica
I20260929 19:00:56.674891 2685 raft_consensus.cc:2229] T 80b7ecf16c99494e8272248d2690d89d P 5fd1996f61cc4be2858c26c0f009154b [term 1 LEADER]: Raft consensus shutting down.
I20260929 19:00:56.674933 2136 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42830:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 19:00:56.675083 2136 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 19:00:56.675112 2684 ts_tablet_manager.cc:1905] T 80b7ecf16c99494e8272248d2690d89d P 3411d1f14ac94407b9fe8e491584772b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 19:00:56.675029 2685 raft_consensus.cc:2258] T 80b7ecf16c99494e8272248d2690d89d P 5fd1996f61cc4be2858c26c0f009154b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 19:00:56.675791 2685 ts_tablet_manager.cc:1905] T 80b7ecf16c99494e8272248d2690d89d P 5fd1996f61cc4be2858c26c0f009154b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 19:00:56.676637 2274 tablet_service.cc:1467] Processing CreateTablet for tablet ba9e2b2c885e43859ff380addea7afc9 (DEFAULT_TABLE table=test-table [id=065c05aa03ff4e44b121694ed8d07415]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:00:56.676753 2274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba9e2b2c885e43859ff380addea7afc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:00:56.676867 2553 tablet_service.cc:1467] Processing CreateTablet for tablet ba9e2b2c885e43859ff380addea7afc9 (DEFAULT_TABLE table=test-table [id=065c05aa03ff4e44b121694ed8d07415]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:00:56.677016 2414 tablet_service.cc:1467] Processing CreateTablet for tablet ba9e2b2c885e43859ff380addea7afc9 (DEFAULT_TABLE table=test-table [id=065c05aa03ff4e44b121694ed8d07415]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:00:56.677124 2414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba9e2b2c885e43859ff380addea7afc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:00:56.677033 2684 ts_tablet_manager.cc:1918] T 80b7ecf16c99494e8272248d2690d89d P 3411d1f14ac94407b9fe8e491584772b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 19:00:56.677227 2684 log.cc:1196] T 80b7ecf16c99494e8272248d2690d89d P 3411d1f14ac94407b9fe8e491584772b: Deleting WAL directory at /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708447806542-392-0/minicluster-data/ts-0/wal/wals/80b7ecf16c99494e8272248d2690d89d
I20260929 19:00:56.677515 2684 ts_tablet_manager.cc:1939] T 80b7ecf16c99494e8272248d2690d89d P 3411d1f14ac94407b9fe8e491584772b: Deleting consensus metadata
I20260929 19:00:56.677923 2122 catalog_manager.cc:4841] TS 3411d1f14ac94407b9fe8e491584772b (127.0.98.1:42221): tablet 80b7ecf16c99494e8272248d2690d89d (table test-table [id=3fe992bba5604cc7a77f241abb38a164]) successfully deleted
I20260929 19:00:56.678385 2643 tablet_bootstrap.cc:492] T ba9e2b2c885e43859ff380addea7afc9 P 0725f0c1e7c74171b33e880e6a6fd5cd: Bootstrap starting.
I20260929 19:00:56.678423 2553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba9e2b2c885e43859ff380addea7afc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:00:56.678797 2643 tablet_bootstrap.cc:654] T ba9e2b2c885e43859ff380addea7afc9 P 0725f0c1e7c74171b33e880e6a6fd5cd: Neither blocks nor log segments found. Creating new log.
I20260929 19:00:56.679490 2643 tablet_bootstrap.cc:492] T ba9e2b2c885e43859ff380addea7afc9 P 0725f0c1e7c74171b33e880e6a6fd5cd: No bootstrap required, opened a new log
I20260929 19:00:56.679548 2643 ts_tablet_manager.cc:1397] T ba9e2b2c885e43859ff380addea7afc9 P 0725f0c1e7c74171b33e880e6a6fd5cd: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260929 19:00:56.679689 2643 raft_consensus.cc:348] T ba9e2b2c885e43859ff380addea7afc9 P 0725f0c1e7c74171b33e880e6a6fd5cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3411d1f14ac94407b9fe8e491584772b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42221 } } peers { permanent_uuid: "0725f0c1e7c74171b33e880e6a6fd5cd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42581 } } peers { permanent_uuid: "5fd1996f61cc4be2858c26c0f009154b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45245 } }
I20260929 19:00:56.679766 2643 raft_consensus.cc:374] T ba9e2b2c885e43859ff380addea7afc9 P 0725f0c1e7c74171b33e880e6a6fd5cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:00:56.679793 2643 raft_consensus.cc:729] T ba9e2b2c885e43859ff380addea7afc9 P 0725f0c1e7c74171b33e880e6a6fd5cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0725f0c1e7c74171b33e880e6a6fd5cd, State: Initialized, Role: FOLLOWER
I20260929 19:00:56.679840 2643 consensus_queue.cc:260] T ba9e2b2c885e43859ff380addea7afc9 P 0725f0c1e7c74171b33e880e6a6fd5cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3411d1f14ac94407b9fe8e491584772b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42221 } } peers { permanent_uuid: "0725f0c1e7c74171b33e880e6a6fd5cd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42581 } } peers { permanent_uuid: "5fd1996f61cc4be2858c26c0f009154b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45245 } }
I20260929 19:00:56.679868 2646 tablet_bootstrap.cc:492] T ba9e2b2c885e43859ff380addea7afc9 P 5fd1996f61cc4be2858c26c0f009154b: Bootstrap starting.
I20260929 19:00:56.679939 2643 ts_tablet_manager.cc:1428] T ba9e2b2c885e43859ff380addea7afc9 P 0725f0c1e7c74171b33e880e6a6fd5cd: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260929 19:00:56.680177 2646 tablet_bootstrap.cc:654] T ba9e2b2c885e43859ff380addea7afc9 P 5fd1996f61cc4be2858c26c0f009154b: Neither blocks nor log segments found. Creating new log.
I20260929 19:00:56.681614 2644 tablet_bootstrap.cc:492] T ba9e2b2c885e43859ff380addea7afc9 P 3411d1f14ac94407b9fe8e491584772b: Bootstrap starting.
I20260929 19:00:56.681811 2646 tablet_bootstrap.cc:492] T ba9e2b2c885e43859ff380addea7afc9 P 5fd1996f61cc4be2858c26c0f009154b: No bootstrap required, opened a new log
I20260929 19:00:56.681864 2646 ts_tablet_manager.cc:1397] T ba9e2b2c885e43859ff380addea7afc9 P 5fd1996f61cc4be2858c26c0f009154b: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260929 19:00:56.682010 2644 tablet_bootstrap.cc:654] T ba9e2b2c885e43859ff380addea7afc9 P 3411d1f14ac94407b9fe8e491584772b: Neither blocks nor log segments found. Creating new log.
I20260929 19:00:56.682116 2646 raft_consensus.cc:348] T ba9e2b2c885e43859ff380addea7afc9 P 5fd1996f61cc4be2858c26c0f009154b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3411d1f14ac94407b9fe8e491584772b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42221 } } peers { permanent_uuid: "0725f0c1e7c74171b33e880e6a6fd5cd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42581 } } peers { permanent_uuid: "5fd1996f61cc4be2858c26c0f009154b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45245 } }
I20260929 19:00:56.682197 2646 raft_consensus.cc:374] T ba9e2b2c885e43859ff380addea7afc9 P 5fd1996f61cc4be2858c26c0f009154b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:00:56.682227 2646 raft_consensus.cc:729] T ba9e2b2c885e43859ff380addea7afc9 P 5fd1996f61cc4be2858c26c0f009154b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fd1996f61cc4be2858c26c0f009154b, State: Initialized, Role: FOLLOWER
I20260929 19:00:56.682276 2646 consensus_queue.cc:260] T ba9e2b2c885e43859ff380addea7afc9 P 5fd1996f61cc4be2858c26c0f009154b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3411d1f14ac94407b9fe8e491584772b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42221 } } peers { permanent_uuid: "0725f0c1e7c74171b33e880e6a6fd5cd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42581 } } peers { permanent_uuid: "5fd1996f61cc4be2858c26c0f009154b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45245 } }
I20260929 19:00:56.682371 2646 ts_tablet_manager.cc:1428] T ba9e2b2c885e43859ff380addea7afc9 P 5fd1996f61cc4be2858c26c0f009154b: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260929 19:00:56.682538 2683 tablet_replica.cc:330] T 80b7ecf16c99494e8272248d2690d89d P 0725f0c1e7c74171b33e880e6a6fd5cd: stopping tablet replica
I20260929 19:00:56.682617 2644 tablet_bootstrap.cc:492] T ba9e2b2c885e43859ff380addea7afc9 P 3411d1f14ac94407b9fe8e491584772b: No bootstrap required, opened a new log
I20260929 19:00:56.682657 2644 ts_tablet_manager.cc:1397] T ba9e2b2c885e43859ff380addea7afc9 P 3411d1f14ac94407b9fe8e491584772b: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260929 19:00:56.682791 2644 raft_consensus.cc:348] T ba9e2b2c885e43859ff380addea7afc9 P 3411d1f14ac94407b9fe8e491584772b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3411d1f14ac94407b9fe8e491584772b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42221 } } peers { permanent_uuid: "0725f0c1e7c74171b33e880e6a6fd5cd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42581 } } peers { permanent_uuid: "5fd1996f61cc4be2858c26c0f009154b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45245 } }
I20260929 19:00:56.682859 2644 raft_consensus.cc:374] T ba9e2b2c885e43859ff380addea7afc9 P 3411d1f14ac94407b9fe8e491584772b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:00:56.682881 2644 raft_consensus.cc:729] T ba9e2b2c885e43859ff380addea7afc9 P 3411d1f14ac94407b9fe8e491584772b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3411d1f14ac94407b9fe8e491584772b, State: Initialized, Role: FOLLOWER
I20260929 19:00:56.683311 2644 consensus_queue.cc:260] T ba9e2b2c885e43859ff380addea7afc9 P 3411d1f14ac94407b9fe8e491584772b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3411d1f14ac94407b9fe8e491584772b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42221 } } peers { permanent_uuid: "0725f0c1e7c74171b33e880e6a6fd5cd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42581 } } peers { permanent_uuid: "5fd1996f61cc4be2858c26c0f009154b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45245 } }
I20260929 19:00:56.683413 2644 ts_tablet_manager.cc:1428] T ba9e2b2c885e43859ff380addea7afc9 P 3411d1f14ac94407b9fe8e491584772b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260929 19:00:56.683498 2683 raft_consensus.cc:2229] T 80b7ecf16c99494e8272248d2690d89d P 0725f0c1e7c74171b33e880e6a6fd5cd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 19:00:56.683557 2683 raft_consensus.cc:2258] T 80b7ecf16c99494e8272248d2690d89d P 0725f0c1e7c74171b33e880e6a6fd5cd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 19:00:56.684113 2685 ts_tablet_manager.cc:1918] T 80b7ecf16c99494e8272248d2690d89d P 5fd1996f61cc4be2858c26c0f009154b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 19:00:56.684202 2685 log.cc:1196] T 80b7ecf16c99494e8272248d2690d89d P 5fd1996f61cc4be2858c26c0f009154b: Deleting WAL directory at /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708447806542-392-0/minicluster-data/ts-2/wal/wals/80b7ecf16c99494e8272248d2690d89d
I20260929 19:00:56.684422 2685 ts_tablet_manager.cc:1939] T 80b7ecf16c99494e8272248d2690d89d P 5fd1996f61cc4be2858c26c0f009154b: Deleting consensus metadata
I20260929 19:00:56.684943 2122 catalog_manager.cc:4841] TS 5fd1996f61cc4be2858c26c0f009154b (127.0.98.3:45245): tablet 80b7ecf16c99494e8272248d2690d89d (table test-table [id=3fe992bba5604cc7a77f241abb38a164]) successfully deleted
I20260929 19:00:56.684962 2683 ts_tablet_manager.cc:1905] T 80b7ecf16c99494e8272248d2690d89d P 0725f0c1e7c74171b33e880e6a6fd5cd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 19:00:56.686084 2683 ts_tablet_manager.cc:1918] T 80b7ecf16c99494e8272248d2690d89d P 0725f0c1e7c74171b33e880e6a6fd5cd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 19:00:56.686168 2683 log.cc:1196] T 80b7ecf16c99494e8272248d2690d89d P 0725f0c1e7c74171b33e880e6a6fd5cd: Deleting WAL directory at /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708447806542-392-0/minicluster-data/ts-1/wal/wals/80b7ecf16c99494e8272248d2690d89d
I20260929 19:00:56.686389 2683 ts_tablet_manager.cc:1939] T 80b7ecf16c99494e8272248d2690d89d P 0725f0c1e7c74171b33e880e6a6fd5cd: Deleting consensus metadata
I20260929 19:00:56.686770 2121 catalog_manager.cc:4841] TS 0725f0c1e7c74171b33e880e6a6fd5cd (127.0.98.2:42581): tablet 80b7ecf16c99494e8272248d2690d89d (table test-table [id=3fe992bba5604cc7a77f241abb38a164]) successfully deleted
I20260929 19:00:56.846450 2647 raft_consensus.cc:482] T ba9e2b2c885e43859ff380addea7afc9 P 3411d1f14ac94407b9fe8e491584772b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 19:00:56.846629 2647 raft_consensus.cc:504] T ba9e2b2c885e43859ff380addea7afc9 P 3411d1f14ac94407b9fe8e491584772b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3411d1f14ac94407b9fe8e491584772b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42221 } } peers { permanent_uuid: "0725f0c1e7c74171b33e880e6a6fd5cd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42581 } } peers { permanent_uuid: "5fd1996f61cc4be2858c26c0f009154b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45245 } }
I20260929 19:00:56.847018 2647 leader_election.cc:288] T ba9e2b2c885e43859ff380addea7afc9 P 3411d1f14ac94407b9fe8e491584772b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0725f0c1e7c74171b33e880e6a6fd5cd (127.0.98.2:42581), 5fd1996f61cc4be2858c26c0f009154b (127.0.98.3:45245)
I20260929 19:00:56.850029 2642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:56.847285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50941 (local address 127.0.98.3:45245)
0929 19:00:56.847340 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0929 19:00:56.847343 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 19:00:56.847417 (+ 74us) server_negotiation.cc:407] Connection header received
0929 19:00:56.847500 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:00:56.847501 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:00:56.847517 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:00:56.847599 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:00:56.847763 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:56.848358 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:56.849328 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:56.849682 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:56.849768 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:56.849779 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0929 19:00:56.849926 (+ 147us) server_negotiation.cc:299] Negotiation successful
0929 19:00:56.849959 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260929 19:00:56.850029 2691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:56.847273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42627 (local address 127.0.98.2:42581)
0929 19:00:56.847403 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0929 19:00:56.847405 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:00:56.847417 (+ 12us) server_negotiation.cc:407] Connection header received
0929 19:00:56.847497 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:00:56.847498 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:00:56.847515 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:00:56.847597 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:00:56.847834 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:56.848293 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:56.849310 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:56.849678 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:56.849755 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:56.849769 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0929 19:00:56.849911 (+ 142us) server_negotiation.cc:299] Negotiation successful
0929 19:00:56.849952 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":49,"threads_started":1}
I20260929 19:00:56.850065 2690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:56.847223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45245 (local address 127.0.98.1:50941)
0929 19:00:56.847347 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0929 19:00:56.847358 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 19:00:56.847432 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:00:56.847606 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:00:56.847607 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:00:56.847609 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:00:56.847700 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:00:56.847702 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:00:56.848444 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:00:56.848445 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:00:56.849232 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:00:56.849234 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:00:56.849781 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:00:56.849782 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:00:56.849840 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:56.849843 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 19:00:56.849912 (+ 69us) client_negotiation.cc:240] Negotiation successful
0929 19:00:56.849951 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":60,"threads_started":1}
I20260929 19:00:56.850068 2689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:56.847224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42581 (local address 127.0.98.1:42627)
0929 19:00:56.847360 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0929 19:00:56.847370 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 19:00:56.847431 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:00:56.847614 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:00:56.847615 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:00:56.847617 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:00:56.847754 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:00:56.847756 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:00:56.848365 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:00:56.848367 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:00:56.849223 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:00:56.849226 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:00:56.849757 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:00:56.849758 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:00:56.849834 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:56.849838 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 19:00:56.849879 (+ 41us) client_negotiation.cc:240] Negotiation successful
0929 19:00:56.849937 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"mutex_wait_us":53,"thread_start_us":69,"threads_started":1}
I20260929 19:00:56.850523 2433 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba9e2b2c885e43859ff380addea7afc9" candidate_uuid: "3411d1f14ac94407b9fe8e491584772b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0725f0c1e7c74171b33e880e6a6fd5cd" is_pre_election: true
I20260929 19:00:56.850517 2574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba9e2b2c885e43859ff380addea7afc9" candidate_uuid: "3411d1f14ac94407b9fe8e491584772b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fd1996f61cc4be2858c26c0f009154b" is_pre_election: true
I20260929 19:00:56.850621 2433 raft_consensus.cc:2454] T ba9e2b2c885e43859ff380addea7afc9 P 0725f0c1e7c74171b33e880e6a6fd5cd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3411d1f14ac94407b9fe8e491584772b in term 0.
I20260929 19:00:56.850636 2574 raft_consensus.cc:2454] T ba9e2b2c885e43859ff380addea7afc9 P 5fd1996f61cc4be2858c26c0f009154b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3411d1f14ac94407b9fe8e491584772b in term 0.
I20260929 19:00:56.850800 2225 leader_election.cc:302] T ba9e2b2c885e43859ff380addea7afc9 P 3411d1f14ac94407b9fe8e491584772b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0725f0c1e7c74171b33e880e6a6fd5cd, 3411d1f14ac94407b9fe8e491584772b; no voters:
I20260929 19:00:56.850955 2647 raft_consensus.cc:2781] T ba9e2b2c885e43859ff380addea7afc9 P 3411d1f14ac94407b9fe8e491584772b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 19:00:56.851006 2647 raft_consensus.cc:482] T ba9e2b2c885e43859ff380addea7afc9 P 3411d1f14ac94407b9fe8e491584772b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 19:00:56.851027 2647 raft_consensus.cc:3037] T ba9e2b2c885e43859ff380addea7afc9 P 3411d1f14ac94407b9fe8e491584772b [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:00:56.851691 2647 raft_consensus.cc:504] T ba9e2b2c885e43859ff380addea7afc9 P 3411d1f14ac94407b9fe8e491584772b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3411d1f14ac94407b9fe8e491584772b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42221 } } peers { permanent_uuid: "0725f0c1e7c74171b33e880e6a6fd5cd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42581 } } peers { permanent_uuid: "5fd1996f61cc4be2858c26c0f009154b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45245 } }
I20260929 19:00:56.851824 2647 leader_election.cc:288] T ba9e2b2c885e43859ff380addea7afc9 P 3411d1f14ac94407b9fe8e491584772b [CANDIDATE]: Term 1 election: Requested vote from peers 0725f0c1e7c74171b33e880e6a6fd5cd (127.0.98.2:42581), 5fd1996f61cc4be2858c26c0f009154b (127.0.98.3:45245)
I20260929 19:00:56.851953 2433 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba9e2b2c885e43859ff380addea7afc9" candidate_uuid: "3411d1f14ac94407b9fe8e491584772b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0725f0c1e7c74171b33e880e6a6fd5cd"
I20260929 19:00:56.851964 2574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba9e2b2c885e43859ff380addea7afc9" candidate_uuid: "3411d1f14ac94407b9fe8e491584772b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fd1996f61cc4be2858c26c0f009154b"
I20260929 19:00:56.852046 2433 raft_consensus.cc:3037] T ba9e2b2c885e43859ff380addea7afc9 P 0725f0c1e7c74171b33e880e6a6fd5cd [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:00:56.852044 2574 raft_consensus.cc:3037] T ba9e2b2c885e43859ff380addea7afc9 P 5fd1996f61cc4be2858c26c0f009154b [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:00:56.852659 2433 raft_consensus.cc:2454] T ba9e2b2c885e43859ff380addea7afc9 P 0725f0c1e7c74171b33e880e6a6fd5cd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3411d1f14ac94407b9fe8e491584772b in term 1.
I20260929 19:00:56.852734 2574 raft_consensus.cc:2454] T ba9e2b2c885e43859ff380addea7afc9 P 5fd1996f61cc4be2858c26c0f009154b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3411d1f14ac94407b9fe8e491584772b in term 1.
I20260929 19:00:56.852895 2225 leader_election.cc:302] T ba9e2b2c885e43859ff380addea7afc9 P 3411d1f14ac94407b9fe8e491584772b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0725f0c1e7c74171b33e880e6a6fd5cd, 3411d1f14ac94407b9fe8e491584772b; no voters:
I20260929 19:00:56.852993 2647 raft_consensus.cc:2781] T ba9e2b2c885e43859ff380addea7afc9 P 3411d1f14ac94407b9fe8e491584772b [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:00:56.853176 2647 raft_consensus.cc:686] T ba9e2b2c885e43859ff380addea7afc9 P 3411d1f14ac94407b9fe8e491584772b [term 1 LEADER]: Becoming Leader. State: Replica: 3411d1f14ac94407b9fe8e491584772b, State: Running, Role: LEADER
I20260929 19:00:56.853257 2647 consensus_queue.cc:237] T ba9e2b2c885e43859ff380addea7afc9 P 3411d1f14ac94407b9fe8e491584772b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3411d1f14ac94407b9fe8e491584772b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42221 } } peers { permanent_uuid: "0725f0c1e7c74171b33e880e6a6fd5cd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42581 } } peers { permanent_uuid: "5fd1996f61cc4be2858c26c0f009154b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45245 } }
I20260929 19:00:56.853995 2136 catalog_manager.cc:5495] T ba9e2b2c885e43859ff380addea7afc9 P 3411d1f14ac94407b9fe8e491584772b reported cstate change: term changed from 0 to 1, leader changed from <none> to 3411d1f14ac94407b9fe8e491584772b (127.0.98.1). New cstate: current_term: 1 leader_uuid: "3411d1f14ac94407b9fe8e491584772b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3411d1f14ac94407b9fe8e491584772b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42221 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0725f0c1e7c74171b33e880e6a6fd5cd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42581 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5fd1996f61cc4be2858c26c0f009154b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45245 } health_report { overall_health: UNKNOWN } } }
I20260929 19:00:56.865713 2694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:56.863208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55224 (local address 127.0.98.1:42221)
0929 19:00:56.863330 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0929 19:00:56.863332 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:00:56.863340 (+ 8us) server_negotiation.cc:407] Connection header received
0929 19:00:56.863385 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:00:56.863385 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:00:56.863403 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:00:56.863474 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 19:00:56.863582 (+ 108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:56.864159 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:56.864914 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:56.865112 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:56.865191 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:56.865374 (+ 183us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 19:00:56.865472 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 19:00:56.865528 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0929 19:00:56.865617 (+ 89us) server_negotiation.cc:299] Negotiation successful
0929 19:00:56.865634 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":84,"threads_started":1}
I20260929 19:00:56.869995 2642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:56.867815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39331 (local address 127.0.98.3:45245)
0929 19:00:56.867853 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0929 19:00:56.867854 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 19:00:56.867988 (+ 134us) server_negotiation.cc:407] Connection header received
0929 19:00:56.868025 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:00:56.868025 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:00:56.868033 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:00:56.868079 (+ 46us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 19:00:56.868219 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:56.868641 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:56.869407 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:56.869581 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:56.869635 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:56.869817 (+ 182us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 19:00:56.869841 (+ 24us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 19:00:56.869879 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0929 19:00:56.869931 (+ 52us) server_negotiation.cc:299] Negotiation successful
0929 19:00:56.869945 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260929 19:00:56.870024 2697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:56.867761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45245 (local address 127.0.98.1:39331)
0929 19:00:56.867915 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0929 19:00:56.867926 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 19:00:56.867985 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:00:56.868113 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:00:56.868113 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:00:56.868115 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 19:00:56.868161 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:00:56.868163 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:00:56.868714 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:00:56.868714 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:00:56.869338 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:00:56.869339 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:00:56.869682 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:00:56.869683 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:00:56.869731 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:56.869743 (+ 12us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 19:00:56.869893 (+ 150us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 19:00:56.869895 (+ 2us) client_negotiation.cc:769] Sending connection context
0929 19:00:56.869931 (+ 36us) client_negotiation.cc:240] Negotiation successful
0929 19:00:56.869961 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":89,"threads_started":1}
I20260929 19:00:56.871937 2433 raft_consensus.cc:1261] T ba9e2b2c885e43859ff380addea7afc9 P 0725f0c1e7c74171b33e880e6a6fd5cd [term 1 FOLLOWER]: Refusing update from remote peer 3411d1f14ac94407b9fe8e491584772b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 19:00:56.871991 2574 raft_consensus.cc:1261] T ba9e2b2c885e43859ff380addea7afc9 P 5fd1996f61cc4be2858c26c0f009154b [term 1 FOLLOWER]: Refusing update from remote peer 3411d1f14ac94407b9fe8e491584772b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 19:00:56.872231 2647 consensus_queue.cc:1035] T ba9e2b2c885e43859ff380addea7afc9 P 3411d1f14ac94407b9fe8e491584772b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5fd1996f61cc4be2858c26c0f009154b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45245 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 19:00:56.872354 2692 consensus_queue.cc:1035] T ba9e2b2c885e43859ff380addea7afc9 P 3411d1f14ac94407b9fe8e491584772b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0725f0c1e7c74171b33e880e6a6fd5cd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 19:00:56.878743 2694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:56.876485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33899 (local address 127.0.98.1:42221)
0929 19:00:56.876579 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0929 19:00:56.876581 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:00:56.876588 (+ 7us) server_negotiation.cc:407] Connection header received
0929 19:00:56.876608 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:00:56.876609 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:00:56.876618 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:00:56.876665 (+ 47us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 19:00:56.876794 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:56.877394 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:56.878042 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:56.878281 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:56.878369 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:56.878504 (+ 135us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 19:00:56.878536 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 19:00:56.878586 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0929 19:00:56.878652 (+ 66us) server_negotiation.cc:299] Negotiation successful
0929 19:00:56.878672 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 19:00:56.878789 2626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:56.876430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42221 (local address 127.0.98.3:33899)
0929 19:00:56.876475 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0929 19:00:56.876486 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 19:00:56.876520 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:00:56.876680 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:00:56.876680 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:00:56.876681 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 19:00:56.876722 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:00:56.876723 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:00:56.877459 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:00:56.877459 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:00:56.877969 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:00:56.877971 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:00:56.878373 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:00:56.878374 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:00:56.878429 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:56.878433 (+ 4us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 19:00:56.878600 (+ 167us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 19:00:56.878602 (+ 2us) client_negotiation.cc:769] Sending connection context
0929 19:00:56.878650 (+ 48us) client_negotiation.cc:240] Negotiation successful
0929 19:00:56.878692 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
W20260929 19:00:56.879107 2669 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260929 19:00:58.397737 2136 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42830:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 19:00:58.397828 2136 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42830:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 19:00:58.398794 2136 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 624b000e6c514bc39d63e63ee983eca5: Sending DeleteTablet for 3 replicas of tablet ba9e2b2c885e43859ff380addea7afc9
I20260929 19:00:58.399068 2274 tablet_service.cc:1514] Processing DeleteTablet for tablet ba9e2b2c885e43859ff380addea7afc9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:00:58 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55212
I20260929 19:00:58.399068 2553 tablet_service.cc:1514] Processing DeleteTablet for tablet ba9e2b2c885e43859ff380addea7afc9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:00:58 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49880
I20260929 19:00:58.399101 2414 tablet_service.cc:1514] Processing DeleteTablet for tablet ba9e2b2c885e43859ff380addea7afc9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:00:58 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53396
I20260929 19:00:58.399261 2724 tablet_replica.cc:330] T ba9e2b2c885e43859ff380addea7afc9 P 3411d1f14ac94407b9fe8e491584772b: stopping tablet replica
I20260929 19:00:58.399322 2724 raft_consensus.cc:2229] T ba9e2b2c885e43859ff380addea7afc9 P 3411d1f14ac94407b9fe8e491584772b [term 1 LEADER]: Raft consensus shutting down.
I20260929 19:00:58.399410 2724 raft_consensus.cc:2258] T ba9e2b2c885e43859ff380addea7afc9 P 3411d1f14ac94407b9fe8e491584772b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 19:00:58.399528 2725 tablet_replica.cc:330] T ba9e2b2c885e43859ff380addea7afc9 P 5fd1996f61cc4be2858c26c0f009154b: stopping tablet replica
I20260929 19:00:58.399588 2725 raft_consensus.cc:2229] T ba9e2b2c885e43859ff380addea7afc9 P 5fd1996f61cc4be2858c26c0f009154b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 19:00:58.399628 2725 raft_consensus.cc:2258] T ba9e2b2c885e43859ff380addea7afc9 P 5fd1996f61cc4be2858c26c0f009154b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 19:00:58.399818 2724 ts_tablet_manager.cc:1905] T ba9e2b2c885e43859ff380addea7afc9 P 3411d1f14ac94407b9fe8e491584772b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 19:00:58.401087 2725 ts_tablet_manager.cc:1905] T ba9e2b2c885e43859ff380addea7afc9 P 5fd1996f61cc4be2858c26c0f009154b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 19:00:58.401891 2724 ts_tablet_manager.cc:1918] T ba9e2b2c885e43859ff380addea7afc9 P 3411d1f14ac94407b9fe8e491584772b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 19:00:58.401954 2724 log.cc:1196] T ba9e2b2c885e43859ff380addea7afc9 P 3411d1f14ac94407b9fe8e491584772b: Deleting WAL directory at /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708447806542-392-0/minicluster-data/ts-0/wal/wals/ba9e2b2c885e43859ff380addea7afc9
I20260929 19:00:58.402173 2724 ts_tablet_manager.cc:1939] T ba9e2b2c885e43859ff380addea7afc9 P 3411d1f14ac94407b9fe8e491584772b: Deleting consensus metadata
I20260929 19:00:58.402581 2122 catalog_manager.cc:4841] TS 3411d1f14ac94407b9fe8e491584772b (127.0.98.1:42221): tablet ba9e2b2c885e43859ff380addea7afc9 (table test-table [id=065c05aa03ff4e44b121694ed8d07415]) successfully deleted
I20260929 19:00:58.402681 2725 ts_tablet_manager.cc:1918] T ba9e2b2c885e43859ff380addea7afc9 P 5fd1996f61cc4be2858c26c0f009154b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 19:00:58.402741 2725 log.cc:1196] T ba9e2b2c885e43859ff380addea7afc9 P 5fd1996f61cc4be2858c26c0f009154b: Deleting WAL directory at /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708447806542-392-0/minicluster-data/ts-2/wal/wals/ba9e2b2c885e43859ff380addea7afc9
I20260929 19:00:58.402946 2725 ts_tablet_manager.cc:1939] T ba9e2b2c885e43859ff380addea7afc9 P 5fd1996f61cc4be2858c26c0f009154b: Deleting consensus metadata
I20260929 19:00:58.403259 2122 catalog_manager.cc:4841] TS 5fd1996f61cc4be2858c26c0f009154b (127.0.98.3:45245): tablet ba9e2b2c885e43859ff380addea7afc9 (table test-table [id=065c05aa03ff4e44b121694ed8d07415]) successfully deleted
I20260929 19:00:58.404289 2726 tablet_replica.cc:330] T ba9e2b2c885e43859ff380addea7afc9 P 0725f0c1e7c74171b33e880e6a6fd5cd: stopping tablet replica
I20260929 19:00:58.404368 2726 raft_consensus.cc:2229] T ba9e2b2c885e43859ff380addea7afc9 P 0725f0c1e7c74171b33e880e6a6fd5cd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 19:00:58.404412 2726 raft_consensus.cc:2258] T ba9e2b2c885e43859ff380addea7afc9 P 0725f0c1e7c74171b33e880e6a6fd5cd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 19:00:58.404626 2726 ts_tablet_manager.cc:1905] T ba9e2b2c885e43859ff380addea7afc9 P 0725f0c1e7c74171b33e880e6a6fd5cd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 19:00:58.406177 2726 ts_tablet_manager.cc:1918] T ba9e2b2c885e43859ff380addea7afc9 P 0725f0c1e7c74171b33e880e6a6fd5cd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 19:00:58.406239 2726 log.cc:1196] T ba9e2b2c885e43859ff380addea7afc9 P 0725f0c1e7c74171b33e880e6a6fd5cd: Deleting WAL directory at /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708447806542-392-0/minicluster-data/ts-1/wal/wals/ba9e2b2c885e43859ff380addea7afc9
I20260929 19:00:58.406430 2726 ts_tablet_manager.cc:1939] T ba9e2b2c885e43859ff380addea7afc9 P 0725f0c1e7c74171b33e880e6a6fd5cd: Deleting consensus metadata
I20260929 19:00:58.406843 2121 catalog_manager.cc:4841] TS 0725f0c1e7c74171b33e880e6a6fd5cd (127.0.98.2:42581): tablet ba9e2b2c885e43859ff380addea7afc9 (table test-table [id=065c05aa03ff4e44b121694ed8d07415]) successfully deleted
Sep 29 19:00:58 dist-test-slave-lqkv krb5kdc[2092](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708458, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 19:00:58 dist-test-slave-lqkv krb5kdc[2092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708458, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 19:00:58.422719 2734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:58.415664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42844 (local address 127.0.98.62:41481)
0929 19:00:58.415865 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0929 19:00:58.415867 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:00:58.415874 (+ 7us) server_negotiation.cc:407] Connection header received
0929 19:00:58.415893 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:00:58.415894 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:00:58.415903 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:00:58.415967 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:00:58.416471 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:58.417037 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:58.417763 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:58.417965 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:58.418019 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:58.420406 (+ 2387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:00:58.420407 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:00:58.420407 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:00:58.420413 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:00:58.421842 (+ 1429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:00:58.422136 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:00:58.422137 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:00:58.422137 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:00:58.422190 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:00:58.422346 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:00:58.422346 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:00:58.422347 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:00:58.422441 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:00:58.422535 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0929 19:00:58.422597 (+ 62us) server_negotiation.cc:299] Negotiation successful
0929 19:00:58.422656 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":108,"threads_started":1}
I20260929 19:00:58.424108 2136 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42844:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 19:00:58.424304 2136 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 19:00:58.425688 2274 tablet_service.cc:1467] Processing CreateTablet for tablet 6d90b3605145486f9339775739e325bb (DEFAULT_TABLE table=test-table [id=3dc45044571748768bb3c3b9176191c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:00:58.425719 2414 tablet_service.cc:1467] Processing CreateTablet for tablet 6d90b3605145486f9339775739e325bb (DEFAULT_TABLE table=test-table [id=3dc45044571748768bb3c3b9176191c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:00:58.425704 2553 tablet_service.cc:1467] Processing CreateTablet for tablet 6d90b3605145486f9339775739e325bb (DEFAULT_TABLE table=test-table [id=3dc45044571748768bb3c3b9176191c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:00:58.425814 2274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d90b3605145486f9339775739e325bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:00:58.425814 2414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d90b3605145486f9339775739e325bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:00:58.425848 2553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d90b3605145486f9339775739e325bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:00:58.426921 2735 tablet_bootstrap.cc:492] T 6d90b3605145486f9339775739e325bb P 5fd1996f61cc4be2858c26c0f009154b: Bootstrap starting.
I20260929 19:00:58.427100 2736 tablet_bootstrap.cc:492] T 6d90b3605145486f9339775739e325bb P 0725f0c1e7c74171b33e880e6a6fd5cd: Bootstrap starting.
I20260929 19:00:58.427296 2737 tablet_bootstrap.cc:492] T 6d90b3605145486f9339775739e325bb P 3411d1f14ac94407b9fe8e491584772b: Bootstrap starting.
I20260929 19:00:58.427452 2735 tablet_bootstrap.cc:654] T 6d90b3605145486f9339775739e325bb P 5fd1996f61cc4be2858c26c0f009154b: Neither blocks nor log segments found. Creating new log.
I20260929 19:00:58.427452 2736 tablet_bootstrap.cc:654] T 6d90b3605145486f9339775739e325bb P 0725f0c1e7c74171b33e880e6a6fd5cd: Neither blocks nor log segments found. Creating new log.
I20260929 19:00:58.427716 2737 tablet_bootstrap.cc:654] T 6d90b3605145486f9339775739e325bb P 3411d1f14ac94407b9fe8e491584772b: Neither blocks nor log segments found. Creating new log.
I20260929 19:00:58.428139 2735 tablet_bootstrap.cc:492] T 6d90b3605145486f9339775739e325bb P 5fd1996f61cc4be2858c26c0f009154b: No bootstrap required, opened a new log
I20260929 19:00:58.428210 2735 ts_tablet_manager.cc:1397] T 6d90b3605145486f9339775739e325bb P 5fd1996f61cc4be2858c26c0f009154b: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260929 19:00:58.428333 2735 raft_consensus.cc:348] T 6d90b3605145486f9339775739e325bb P 5fd1996f61cc4be2858c26c0f009154b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3411d1f14ac94407b9fe8e491584772b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42221 } } peers { permanent_uuid: "0725f0c1e7c74171b33e880e6a6fd5cd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42581 } } peers { permanent_uuid: "5fd1996f61cc4be2858c26c0f009154b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45245 } }
I20260929 19:00:58.428397 2736 tablet_bootstrap.cc:492] T 6d90b3605145486f9339775739e325bb P 0725f0c1e7c74171b33e880e6a6fd5cd: No bootstrap required, opened a new log
I20260929 19:00:58.428409 2735 raft_consensus.cc:374] T 6d90b3605145486f9339775739e325bb P 5fd1996f61cc4be2858c26c0f009154b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:00:58.428432 2735 raft_consensus.cc:729] T 6d90b3605145486f9339775739e325bb P 5fd1996f61cc4be2858c26c0f009154b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fd1996f61cc4be2858c26c0f009154b, State: Initialized, Role: FOLLOWER
I20260929 19:00:58.428442 2736 ts_tablet_manager.cc:1397] T 6d90b3605145486f9339775739e325bb P 0725f0c1e7c74171b33e880e6a6fd5cd: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260929 19:00:58.428514 2735 consensus_queue.cc:260] T 6d90b3605145486f9339775739e325bb P 5fd1996f61cc4be2858c26c0f009154b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3411d1f14ac94407b9fe8e491584772b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42221 } } peers { permanent_uuid: "0725f0c1e7c74171b33e880e6a6fd5cd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42581 } } peers { permanent_uuid: "5fd1996f61cc4be2858c26c0f009154b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45245 } }
I20260929 19:00:58.428629 2735 ts_tablet_manager.cc:1428] T 6d90b3605145486f9339775739e325bb P 5fd1996f61cc4be2858c26c0f009154b: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260929 19:00:58.428629 2736 raft_consensus.cc:348] T 6d90b3605145486f9339775739e325bb P 0725f0c1e7c74171b33e880e6a6fd5cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3411d1f14ac94407b9fe8e491584772b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42221 } } peers { permanent_uuid: "0725f0c1e7c74171b33e880e6a6fd5cd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42581 } } peers { permanent_uuid: "5fd1996f61cc4be2858c26c0f009154b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45245 } }
I20260929 19:00:58.428695 2736 raft_consensus.cc:374] T 6d90b3605145486f9339775739e325bb P 0725f0c1e7c74171b33e880e6a6fd5cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:00:58.428717 2736 raft_consensus.cc:729] T 6d90b3605145486f9339775739e325bb P 0725f0c1e7c74171b33e880e6a6fd5cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0725f0c1e7c74171b33e880e6a6fd5cd, State: Initialized, Role: FOLLOWER
I20260929 19:00:58.428782 2736 consensus_queue.cc:260] T 6d90b3605145486f9339775739e325bb P 0725f0c1e7c74171b33e880e6a6fd5cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3411d1f14ac94407b9fe8e491584772b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42221 } } peers { permanent_uuid: "0725f0c1e7c74171b33e880e6a6fd5cd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42581 } } peers { permanent_uuid: "5fd1996f61cc4be2858c26c0f009154b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45245 } }
I20260929 19:00:58.429016 2737 tablet_bootstrap.cc:492] T 6d90b3605145486f9339775739e325bb P 3411d1f14ac94407b9fe8e491584772b: No bootstrap required, opened a new log
I20260929 19:00:58.429064 2737 ts_tablet_manager.cc:1397] T 6d90b3605145486f9339775739e325bb P 3411d1f14ac94407b9fe8e491584772b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 19:00:58.429126 2736 ts_tablet_manager.cc:1428] T 6d90b3605145486f9339775739e325bb P 0725f0c1e7c74171b33e880e6a6fd5cd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 19:00:58.429167 2737 raft_consensus.cc:348] T 6d90b3605145486f9339775739e325bb P 3411d1f14ac94407b9fe8e491584772b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3411d1f14ac94407b9fe8e491584772b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42221 } } peers { permanent_uuid: "0725f0c1e7c74171b33e880e6a6fd5cd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42581 } } peers { permanent_uuid: "5fd1996f61cc4be2858c26c0f009154b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45245 } }
I20260929 19:00:58.429243 2737 raft_consensus.cc:374] T 6d90b3605145486f9339775739e325bb P 3411d1f14ac94407b9fe8e491584772b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:00:58.429271 2737 raft_consensus.cc:729] T 6d90b3605145486f9339775739e325bb P 3411d1f14ac94407b9fe8e491584772b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3411d1f14ac94407b9fe8e491584772b, State: Initialized, Role: FOLLOWER
I20260929 19:00:58.429318 2737 consensus_queue.cc:260] T 6d90b3605145486f9339775739e325bb P 3411d1f14ac94407b9fe8e491584772b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3411d1f14ac94407b9fe8e491584772b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42221 } } peers { permanent_uuid: "0725f0c1e7c74171b33e880e6a6fd5cd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42581 } } peers { permanent_uuid: "5fd1996f61cc4be2858c26c0f009154b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45245 } }
I20260929 19:00:58.429409 2737 ts_tablet_manager.cc:1428] T 6d90b3605145486f9339775739e325bb P 3411d1f14ac94407b9fe8e491584772b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260929 19:00:58.474236 2741 raft_consensus.cc:482] T 6d90b3605145486f9339775739e325bb P 0725f0c1e7c74171b33e880e6a6fd5cd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 19:00:58.474323 2741 raft_consensus.cc:504] T 6d90b3605145486f9339775739e325bb P 0725f0c1e7c74171b33e880e6a6fd5cd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3411d1f14ac94407b9fe8e491584772b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42221 } } peers { permanent_uuid: "0725f0c1e7c74171b33e880e6a6fd5cd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42581 } } peers { permanent_uuid: "5fd1996f61cc4be2858c26c0f009154b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45245 } }
I20260929 19:00:58.474815 2741 leader_election.cc:288] T 6d90b3605145486f9339775739e325bb P 0725f0c1e7c74171b33e880e6a6fd5cd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3411d1f14ac94407b9fe8e491584772b (127.0.98.1:42221), 5fd1996f61cc4be2858c26c0f009154b (127.0.98.3:45245)
I20260929 19:00:58.478943 2745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:58.475317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50585 (local address 127.0.98.3:45245)
0929 19:00:58.475463 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0929 19:00:58.475465 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:00:58.475472 (+ 7us) server_negotiation.cc:407] Connection header received
0929 19:00:58.475502 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:00:58.475502 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:00:58.475510 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:00:58.475591 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:00:58.475722 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:58.476419 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:58.478008 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:58.478573 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:58.478653 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:58.478664 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0929 19:00:58.478828 (+ 164us) server_negotiation.cc:299] Negotiation successful
0929 19:00:58.478866 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":88,"threads_started":1}
I20260929 19:00:58.478936 2743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:58.475040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48241 (local address 127.0.98.1:42221)
0929 19:00:58.475203 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0929 19:00:58.475205 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:00:58.475217 (+ 12us) server_negotiation.cc:407] Connection header received
0929 19:00:58.475303 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:00:58.475304 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:00:58.475316 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:00:58.475402 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:00:58.475572 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:58.476381 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:58.478008 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:58.478590 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:58.478653 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:58.478664 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0929 19:00:58.478824 (+ 160us) server_negotiation.cc:299] Negotiation successful
0929 19:00:58.478863 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":109,"threads_started":1}
I20260929 19:00:58.478969 2744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:58.475239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45245 (local address 127.0.98.2:50585)
0929 19:00:58.475334 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0929 19:00:58.475343 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 19:00:58.475406 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:00:58.475604 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:00:58.475604 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:00:58.475606 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:00:58.475655 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:00:58.475656 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:00:58.476511 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:00:58.476511 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:00:58.477908 (+ 1397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:00:58.477911 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:00:58.478665 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:00:58.478666 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:00:58.478745 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:58.478748 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 19:00:58.478806 (+ 58us) client_negotiation.cc:240] Negotiation successful
0929 19:00:58.478864 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80,"mutex_wait_us":83,"thread_start_us":68,"threads_started":1}
I20260929 19:00:58.478972 2742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:58.475007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42221 (local address 127.0.98.2:48241)
0929 19:00:58.475172 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0929 19:00:58.475184 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 19:00:58.475234 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:00:58.475417 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:00:58.475418 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:00:58.475420 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:00:58.475492 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:00:58.475494 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:00:58.476510 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:00:58.476511 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:00:58.477908 (+ 1397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:00:58.477911 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:00:58.478663 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:00:58.478665 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:00:58.478749 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:58.478752 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 19:00:58.478806 (+ 54us) client_negotiation.cc:240] Negotiation successful
0929 19:00:58.478864 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"mutex_wait_us":86,"thread_start_us":119,"threads_started":1}
I20260929 19:00:58.479343 2574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d90b3605145486f9339775739e325bb" candidate_uuid: "0725f0c1e7c74171b33e880e6a6fd5cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fd1996f61cc4be2858c26c0f009154b" is_pre_election: true
I20260929 19:00:58.479431 2574 raft_consensus.cc:2454] T 6d90b3605145486f9339775739e325bb P 5fd1996f61cc4be2858c26c0f009154b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0725f0c1e7c74171b33e880e6a6fd5cd in term 0.
I20260929 19:00:58.479620 2366 leader_election.cc:302] T 6d90b3605145486f9339775739e325bb P 0725f0c1e7c74171b33e880e6a6fd5cd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0725f0c1e7c74171b33e880e6a6fd5cd, 5fd1996f61cc4be2858c26c0f009154b; no voters:
I20260929 19:00:58.479678 2294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d90b3605145486f9339775739e325bb" candidate_uuid: "0725f0c1e7c74171b33e880e6a6fd5cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3411d1f14ac94407b9fe8e491584772b" is_pre_election: true
I20260929 19:00:58.479740 2741 raft_consensus.cc:2781] T 6d90b3605145486f9339775739e325bb P 0725f0c1e7c74171b33e880e6a6fd5cd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 19:00:58.479774 2741 raft_consensus.cc:482] T 6d90b3605145486f9339775739e325bb P 0725f0c1e7c74171b33e880e6a6fd5cd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 19:00:58.479773 2294 raft_consensus.cc:2454] T 6d90b3605145486f9339775739e325bb P 3411d1f14ac94407b9fe8e491584772b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0725f0c1e7c74171b33e880e6a6fd5cd in term 0.
I20260929 19:00:58.479803 2741 raft_consensus.cc:3037] T 6d90b3605145486f9339775739e325bb P 0725f0c1e7c74171b33e880e6a6fd5cd [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:00:58.480538 2741 raft_consensus.cc:504] T 6d90b3605145486f9339775739e325bb P 0725f0c1e7c74171b33e880e6a6fd5cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3411d1f14ac94407b9fe8e491584772b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42221 } } peers { permanent_uuid: "0725f0c1e7c74171b33e880e6a6fd5cd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42581 } } peers { permanent_uuid: "5fd1996f61cc4be2858c26c0f009154b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45245 } }
I20260929 19:00:58.480659 2741 leader_election.cc:288] T 6d90b3605145486f9339775739e325bb P 0725f0c1e7c74171b33e880e6a6fd5cd [CANDIDATE]: Term 1 election: Requested vote from peers 3411d1f14ac94407b9fe8e491584772b (127.0.98.1:42221), 5fd1996f61cc4be2858c26c0f009154b (127.0.98.3:45245)
I20260929 19:00:58.480851 2294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d90b3605145486f9339775739e325bb" candidate_uuid: "0725f0c1e7c74171b33e880e6a6fd5cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3411d1f14ac94407b9fe8e491584772b"
I20260929 19:00:58.480888 2574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d90b3605145486f9339775739e325bb" candidate_uuid: "0725f0c1e7c74171b33e880e6a6fd5cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fd1996f61cc4be2858c26c0f009154b"
I20260929 19:00:58.480935 2294 raft_consensus.cc:3037] T 6d90b3605145486f9339775739e325bb P 3411d1f14ac94407b9fe8e491584772b [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:00:58.480945 2574 raft_consensus.cc:3037] T 6d90b3605145486f9339775739e325bb P 5fd1996f61cc4be2858c26c0f009154b [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:00:58.481640 2574 raft_consensus.cc:2454] T 6d90b3605145486f9339775739e325bb P 5fd1996f61cc4be2858c26c0f009154b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0725f0c1e7c74171b33e880e6a6fd5cd in term 1.
I20260929 19:00:58.481640 2294 raft_consensus.cc:2454] T 6d90b3605145486f9339775739e325bb P 3411d1f14ac94407b9fe8e491584772b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0725f0c1e7c74171b33e880e6a6fd5cd in term 1.
I20260929 19:00:58.481794 2366 leader_election.cc:302] T 6d90b3605145486f9339775739e325bb P 0725f0c1e7c74171b33e880e6a6fd5cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0725f0c1e7c74171b33e880e6a6fd5cd, 3411d1f14ac94407b9fe8e491584772b; no voters:
I20260929 19:00:58.481884 2741 raft_consensus.cc:2781] T 6d90b3605145486f9339775739e325bb P 0725f0c1e7c74171b33e880e6a6fd5cd [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:00:58.482024 2741 raft_consensus.cc:686] T 6d90b3605145486f9339775739e325bb P 0725f0c1e7c74171b33e880e6a6fd5cd [term 1 LEADER]: Becoming Leader. State: Replica: 0725f0c1e7c74171b33e880e6a6fd5cd, State: Running, Role: LEADER
I20260929 19:00:58.482077 2741 consensus_queue.cc:237] T 6d90b3605145486f9339775739e325bb P 0725f0c1e7c74171b33e880e6a6fd5cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3411d1f14ac94407b9fe8e491584772b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42221 } } peers { permanent_uuid: "0725f0c1e7c74171b33e880e6a6fd5cd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42581 } } peers { permanent_uuid: "5fd1996f61cc4be2858c26c0f009154b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45245 } }
I20260929 19:00:58.482738 2137 catalog_manager.cc:5495] T 6d90b3605145486f9339775739e325bb P 0725f0c1e7c74171b33e880e6a6fd5cd reported cstate change: term changed from 0 to 1, leader changed from <none> to 0725f0c1e7c74171b33e880e6a6fd5cd (127.0.98.2). New cstate: current_term: 1 leader_uuid: "0725f0c1e7c74171b33e880e6a6fd5cd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3411d1f14ac94407b9fe8e491584772b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0725f0c1e7c74171b33e880e6a6fd5cd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42581 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5fd1996f61cc4be2858c26c0f009154b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45245 } health_report { overall_health: UNKNOWN } } }
I20260929 19:00:58.504290 2749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:58.501851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53412 (local address 127.0.98.2:42581)
0929 19:00:58.502000 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0929 19:00:58.502002 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:00:58.502009 (+ 7us) server_negotiation.cc:407] Connection header received
0929 19:00:58.502041 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:00:58.502042 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:00:58.502060 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:00:58.502132 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 19:00:58.502332 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:58.502808 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:58.503575 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:58.503781 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:58.503836 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:58.503945 (+ 109us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 19:00:58.504045 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 19:00:58.504095 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0929 19:00:58.504132 (+ 37us) server_negotiation.cc:299] Negotiation successful
0929 19:00:58.504146 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":82,"threads_started":1}
I20260929 19:00:58.508946 2745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:58.506777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45539 (local address 127.0.98.3:45245)
0929 19:00:58.506825 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0929 19:00:58.506827 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:00:58.506879 (+ 52us) server_negotiation.cc:407] Connection header received
0929 19:00:58.506917 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:00:58.506918 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:00:58.506927 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:00:58.506985 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 19:00:58.507089 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:58.507537 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:58.508273 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:58.508466 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:58.508533 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:58.508727 (+ 194us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 19:00:58.508754 (+ 27us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 19:00:58.508846 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0929 19:00:58.508871 (+ 25us) server_negotiation.cc:299] Negotiation successful
0929 19:00:58.508896 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260929 19:00:58.508951 2752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:58.506639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45245 (local address 127.0.98.2:45539)
0929 19:00:58.506809 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0929 19:00:58.506820 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 19:00:58.506879 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:00:58.506988 (+ 109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:00:58.506989 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:00:58.506993 (+ 4us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 19:00:58.507038 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:00:58.507039 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:00:58.507612 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:00:58.507613 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:00:58.508171 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:00:58.508172 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:00:58.508558 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:00:58.508559 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:00:58.508622 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:58.508634 (+ 12us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 19:00:58.508827 (+ 193us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 19:00:58.508829 (+ 2us) client_negotiation.cc:769] Sending connection context
0929 19:00:58.508854 (+ 25us) client_negotiation.cc:240] Negotiation successful
0929 19:00:58.508889 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":105,"threads_started":1}
I20260929 19:00:58.510895 2574 raft_consensus.cc:1261] T 6d90b3605145486f9339775739e325bb P 5fd1996f61cc4be2858c26c0f009154b [term 1 FOLLOWER]: Refusing update from remote peer 0725f0c1e7c74171b33e880e6a6fd5cd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 19:00:58.511082 2294 raft_consensus.cc:1261] T 6d90b3605145486f9339775739e325bb P 3411d1f14ac94407b9fe8e491584772b [term 1 FOLLOWER]: Refusing update from remote peer 0725f0c1e7c74171b33e880e6a6fd5cd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 19:00:58.511166 2741 consensus_queue.cc:1035] T 6d90b3605145486f9339775739e325bb P 0725f0c1e7c74171b33e880e6a6fd5cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "5fd1996f61cc4be2858c26c0f009154b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45245 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 19:00:58.511304 2746 consensus_queue.cc:1035] T 6d90b3605145486f9339775739e325bb P 0725f0c1e7c74171b33e880e6a6fd5cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "3411d1f14ac94407b9fe8e491584772b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 19:00:58.518555 2757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:58.515713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42581 (local address 127.0.98.3:56361)
0929 19:00:58.515846 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0929 19:00:58.515856 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 19:00:58.515910 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:00:58.516302 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:00:58.516303 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:00:58.516305 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 19:00:58.516340 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:00:58.516341 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:00:58.517056 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:00:58.517057 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:00:58.517700 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:00:58.517702 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:00:58.518165 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:00:58.518166 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:00:58.518206 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:58.518210 (+ 4us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 19:00:58.518398 (+ 188us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 19:00:58.518400 (+ 2us) client_negotiation.cc:769] Sending connection context
0929 19:00:58.518445 (+ 45us) client_negotiation.cc:240] Negotiation successful
0929 19:00:58.518486 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":75,"threads_started":1}
I20260929 19:00:58.518544 2749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:58.515775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56361 (local address 127.0.98.2:42581)
0929 19:00:58.515810 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0929 19:00:58.515812 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:00:58.515917 (+ 105us) server_negotiation.cc:407] Connection header received
0929 19:00:58.515988 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:00:58.515989 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:00:58.516000 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:00:58.516238 (+ 238us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 19:00:58.516424 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:58.516977 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:58.517782 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:58.518068 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:58.518131 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:58.518301 (+ 170us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 19:00:58.518335 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 19:00:58.518387 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0929 19:00:58.518457 (+ 70us) server_negotiation.cc:299] Negotiation successful
0929 19:00:58.518477 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260929 19:00:58.524971 2136 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42844:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 19:00:58.525066 2136 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42844:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 19:00:58.525637 2136 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 624b000e6c514bc39d63e63ee983eca5: Sending DeleteTablet for 3 replicas of tablet 6d90b3605145486f9339775739e325bb
I20260929 19:00:58.525888 2553 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d90b3605145486f9339775739e325bb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:00:58 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49880
I20260929 19:00:58.525892 2274 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d90b3605145486f9339775739e325bb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:00:58 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55212
I20260929 19:00:58.525986 2725 tablet_replica.cc:330] T 6d90b3605145486f9339775739e325bb P 5fd1996f61cc4be2858c26c0f009154b: stopping tablet replica
I20260929 19:00:58.525995 2724 tablet_replica.cc:330] T 6d90b3605145486f9339775739e325bb P 3411d1f14ac94407b9fe8e491584772b: stopping tablet replica
I20260929 19:00:58.526039 2725 raft_consensus.cc:2229] T 6d90b3605145486f9339775739e325bb P 5fd1996f61cc4be2858c26c0f009154b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 19:00:58.526044 2724 raft_consensus.cc:2229] T 6d90b3605145486f9339775739e325bb P 3411d1f14ac94407b9fe8e491584772b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 19:00:58.526075 2724 raft_consensus.cc:2258] T 6d90b3605145486f9339775739e325bb P 3411d1f14ac94407b9fe8e491584772b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 19:00:58.526073 2725 raft_consensus.cc:2258] T 6d90b3605145486f9339775739e325bb P 5fd1996f61cc4be2858c26c0f009154b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 19:00:58.526389 2724 ts_tablet_manager.cc:1905] T 6d90b3605145486f9339775739e325bb P 3411d1f14ac94407b9fe8e491584772b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 19:00:58.526490 2725 ts_tablet_manager.cc:1905] T 6d90b3605145486f9339775739e325bb P 5fd1996f61cc4be2858c26c0f009154b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 19:00:58.527702 2414 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d90b3605145486f9339775739e325bb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:00:58 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53396
I20260929 19:00:58.527835 2136 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42844:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 19:00:58.527999 2136 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 19:00:58.528026 2724 ts_tablet_manager.cc:1918] T 6d90b3605145486f9339775739e325bb P 3411d1f14ac94407b9fe8e491584772b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 19:00:58.528093 2724 log.cc:1196] T 6d90b3605145486f9339775739e325bb P 3411d1f14ac94407b9fe8e491584772b: Deleting WAL directory at /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708447806542-392-0/minicluster-data/ts-0/wal/wals/6d90b3605145486f9339775739e325bb
I20260929 19:00:58.528312 2724 ts_tablet_manager.cc:1939] T 6d90b3605145486f9339775739e325bb P 3411d1f14ac94407b9fe8e491584772b: Deleting consensus metadata
I20260929 19:00:58.528442 2726 tablet_replica.cc:330] T 6d90b3605145486f9339775739e325bb P 0725f0c1e7c74171b33e880e6a6fd5cd: stopping tablet replica
I20260929 19:00:58.528519 2726 raft_consensus.cc:2229] T 6d90b3605145486f9339775739e325bb P 0725f0c1e7c74171b33e880e6a6fd5cd [term 1 LEADER]: Raft consensus shutting down.
I20260929 19:00:58.528620 2726 raft_consensus.cc:2258] T 6d90b3605145486f9339775739e325bb P 0725f0c1e7c74171b33e880e6a6fd5cd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 19:00:58.529066 2726 ts_tablet_manager.cc:1905] T 6d90b3605145486f9339775739e325bb P 0725f0c1e7c74171b33e880e6a6fd5cd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 19:00:58.529605 2274 tablet_service.cc:1467] Processing CreateTablet for tablet 31a772fb86714054ba38e1a099c7fedf (DEFAULT_TABLE table=test-table [id=38f354c0b6bb4c8fa1cc44db19d50817]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:00:58.529604 2553 tablet_service.cc:1467] Processing CreateTablet for tablet 31a772fb86714054ba38e1a099c7fedf (DEFAULT_TABLE table=test-table [id=38f354c0b6bb4c8fa1cc44db19d50817]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:00:58.529709 2274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31a772fb86714054ba38e1a099c7fedf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:00:58.529776 2414 tablet_service.cc:1467] Processing CreateTablet for tablet 31a772fb86714054ba38e1a099c7fedf (DEFAULT_TABLE table=test-table [id=38f354c0b6bb4c8fa1cc44db19d50817]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:00:58.529872 2414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31a772fb86714054ba38e1a099c7fedf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:00:58.530014 2725 ts_tablet_manager.cc:1918] T 6d90b3605145486f9339775739e325bb P 5fd1996f61cc4be2858c26c0f009154b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 19:00:58.530064 2725 log.cc:1196] T 6d90b3605145486f9339775739e325bb P 5fd1996f61cc4be2858c26c0f009154b: Deleting WAL directory at /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708447806542-392-0/minicluster-data/ts-2/wal/wals/6d90b3605145486f9339775739e325bb
I20260929 19:00:58.530287 2725 ts_tablet_manager.cc:1939] T 6d90b3605145486f9339775739e325bb P 5fd1996f61cc4be2858c26c0f009154b: Deleting consensus metadata
I20260929 19:00:58.530413 2553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31a772fb86714054ba38e1a099c7fedf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:00:58.531054 2737 tablet_bootstrap.cc:492] T 31a772fb86714054ba38e1a099c7fedf P 3411d1f14ac94407b9fe8e491584772b: Bootstrap starting.
I20260929 19:00:58.531327 2737 tablet_bootstrap.cc:654] T 31a772fb86714054ba38e1a099c7fedf P 3411d1f14ac94407b9fe8e491584772b: Neither blocks nor log segments found. Creating new log.
I20260929 19:00:58.531598 2736 tablet_bootstrap.cc:492] T 31a772fb86714054ba38e1a099c7fedf P 0725f0c1e7c74171b33e880e6a6fd5cd: Bootstrap starting.
I20260929 19:00:58.531592 2735 tablet_bootstrap.cc:492] T 31a772fb86714054ba38e1a099c7fedf P 5fd1996f61cc4be2858c26c0f009154b: Bootstrap starting.
I20260929 19:00:58.531908 2735 tablet_bootstrap.cc:654] T 31a772fb86714054ba38e1a099c7fedf P 5fd1996f61cc4be2858c26c0f009154b: Neither blocks nor log segments found. Creating new log.
I20260929 19:00:58.531992 2736 tablet_bootstrap.cc:654] T 31a772fb86714054ba38e1a099c7fedf P 0725f0c1e7c74171b33e880e6a6fd5cd: Neither blocks nor log segments found. Creating new log.
I20260929 19:00:58.532294 2726 ts_tablet_manager.cc:1918] T 6d90b3605145486f9339775739e325bb P 0725f0c1e7c74171b33e880e6a6fd5cd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 19:00:58.532356 2726 log.cc:1196] T 6d90b3605145486f9339775739e325bb P 0725f0c1e7c74171b33e880e6a6fd5cd: Deleting WAL directory at /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708447806542-392-0/minicluster-data/ts-1/wal/wals/6d90b3605145486f9339775739e325bb
I20260929 19:00:58.532585 2726 ts_tablet_manager.cc:1939] T 6d90b3605145486f9339775739e325bb P 0725f0c1e7c74171b33e880e6a6fd5cd: Deleting consensus metadata
I20260929 19:00:58.532620 2737 tablet_bootstrap.cc:492] T 31a772fb86714054ba38e1a099c7fedf P 3411d1f14ac94407b9fe8e491584772b: No bootstrap required, opened a new log
I20260929 19:00:58.532680 2737 ts_tablet_manager.cc:1397] T 31a772fb86714054ba38e1a099c7fedf P 3411d1f14ac94407b9fe8e491584772b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 19:00:58.532793 2737 raft_consensus.cc:348] T 31a772fb86714054ba38e1a099c7fedf P 3411d1f14ac94407b9fe8e491584772b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3411d1f14ac94407b9fe8e491584772b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42221 } } peers { permanent_uuid: "0725f0c1e7c74171b33e880e6a6fd5cd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42581 } } peers { permanent_uuid: "5fd1996f61cc4be2858c26c0f009154b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45245 } }
I20260929 19:00:58.532892 2737 raft_consensus.cc:374] T 31a772fb86714054ba38e1a099c7fedf P 3411d1f14ac94407b9fe8e491584772b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:00:58.532915 2737 raft_consensus.cc:729] T 31a772fb86714054ba38e1a099c7fedf P 3411d1f14ac94407b9fe8e491584772b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3411d1f14ac94407b9fe8e491584772b, State: Initialized, Role: FOLLOWER
I20260929 19:00:58.532910 2121 catalog_manager.cc:4841] TS 0725f0c1e7c74171b33e880e6a6fd5cd (127.0.98.2:42581): tablet 6d90b3605145486f9339775739e325bb (table test-table [id=3dc45044571748768bb3c3b9176191c2]) successfully deleted
I20260929 19:00:58.532933 2736 tablet_bootstrap.cc:492] T 31a772fb86714054ba38e1a099c7fedf P 0725f0c1e7c74171b33e880e6a6fd5cd: No bootstrap required, opened a new log
I20260929 19:00:58.532956 2737 consensus_queue.cc:260] T 31a772fb86714054ba38e1a099c7fedf P 3411d1f14ac94407b9fe8e491584772b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3411d1f14ac94407b9fe8e491584772b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42221 } } peers { permanent_uuid: "0725f0c1e7c74171b33e880e6a6fd5cd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42581 } } peers { permanent_uuid: "5fd1996f61cc4be2858c26c0f009154b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45245 } }
I20260929 19:00:58.532985 2736 ts_tablet_manager.cc:1397] T 31a772fb86714054ba38e1a099c7fedf P 0725f0c1e7c74171b33e880e6a6fd5cd: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260929 19:00:58.533053 2737 ts_tablet_manager.cc:1428] T 31a772fb86714054ba38e1a099c7fedf P 3411d1f14ac94407b9fe8e491584772b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260929 19:00:58.533133 2736 raft_consensus.cc:348] T 31a772fb86714054ba38e1a099c7fedf P 0725f0c1e7c74171b33e880e6a6fd5cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3411d1f14ac94407b9fe8e491584772b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42221 } } peers { permanent_uuid: "0725f0c1e7c74171b33e880e6a6fd5cd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42581 } } peers { permanent_uuid: "5fd1996f61cc4be2858c26c0f009154b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45245 } }
I20260929 19:00:58.533187 2736 raft_consensus.cc:374] T 31a772fb86714054ba38e1a099c7fedf P 0725f0c1e7c74171b33e880e6a6fd5cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:00:58.533192 2122 catalog_manager.cc:4841] TS 5fd1996f61cc4be2858c26c0f009154b (127.0.98.3:45245): tablet 6d90b3605145486f9339775739e325bb (table test-table [id=3dc45044571748768bb3c3b9176191c2]) successfully deleted
I20260929 19:00:58.533215 2736 raft_consensus.cc:729] T 31a772fb86714054ba38e1a099c7fedf P 0725f0c1e7c74171b33e880e6a6fd5cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0725f0c1e7c74171b33e880e6a6fd5cd, State: Initialized, Role: FOLLOWER
I20260929 19:00:58.533258 2736 consensus_queue.cc:260] T 31a772fb86714054ba38e1a099c7fedf P 0725f0c1e7c74171b33e880e6a6fd5cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3411d1f14ac94407b9fe8e491584772b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42221 } } peers { permanent_uuid: "0725f0c1e7c74171b33e880e6a6fd5cd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42581 } } peers { permanent_uuid: "5fd1996f61cc4be2858c26c0f009154b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45245 } }
I20260929 19:00:58.533380 2736 ts_tablet_manager.cc:1428] T 31a772fb86714054ba38e1a099c7fedf P 0725f0c1e7c74171b33e880e6a6fd5cd: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260929 19:00:58.533654 2122 catalog_manager.cc:4841] TS 3411d1f14ac94407b9fe8e491584772b (127.0.98.1:42221): tablet 6d90b3605145486f9339775739e325bb (table test-table [id=3dc45044571748768bb3c3b9176191c2]) successfully deleted
I20260929 19:00:58.533900 2735 tablet_bootstrap.cc:492] T 31a772fb86714054ba38e1a099c7fedf P 5fd1996f61cc4be2858c26c0f009154b: No bootstrap required, opened a new log
I20260929 19:00:58.533960 2735 ts_tablet_manager.cc:1397] T 31a772fb86714054ba38e1a099c7fedf P 5fd1996f61cc4be2858c26c0f009154b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 19:00:58.534065 2735 raft_consensus.cc:348] T 31a772fb86714054ba38e1a099c7fedf P 5fd1996f61cc4be2858c26c0f009154b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3411d1f14ac94407b9fe8e491584772b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42221 } } peers { permanent_uuid: "0725f0c1e7c74171b33e880e6a6fd5cd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42581 } } peers { permanent_uuid: "5fd1996f61cc4be2858c26c0f009154b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45245 } }
I20260929 19:00:58.534152 2735 raft_consensus.cc:374] T 31a772fb86714054ba38e1a099c7fedf P 5fd1996f61cc4be2858c26c0f009154b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:00:58.534179 2735 raft_consensus.cc:729] T 31a772fb86714054ba38e1a099c7fedf P 5fd1996f61cc4be2858c26c0f009154b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fd1996f61cc4be2858c26c0f009154b, State: Initialized, Role: FOLLOWER
I20260929 19:00:58.534237 2735 consensus_queue.cc:260] T 31a772fb86714054ba38e1a099c7fedf P 5fd1996f61cc4be2858c26c0f009154b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3411d1f14ac94407b9fe8e491584772b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42221 } } peers { permanent_uuid: "0725f0c1e7c74171b33e880e6a6fd5cd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42581 } } peers { permanent_uuid: "5fd1996f61cc4be2858c26c0f009154b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45245 } }
I20260929 19:00:58.534333 2735 ts_tablet_manager.cc:1428] T 31a772fb86714054ba38e1a099c7fedf P 5fd1996f61cc4be2858c26c0f009154b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260929 19:00:58.623926 2700 raft_consensus.cc:482] T 31a772fb86714054ba38e1a099c7fedf P 3411d1f14ac94407b9fe8e491584772b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 19:00:58.624033 2700 raft_consensus.cc:504] T 31a772fb86714054ba38e1a099c7fedf P 3411d1f14ac94407b9fe8e491584772b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3411d1f14ac94407b9fe8e491584772b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42221 } } peers { permanent_uuid: "0725f0c1e7c74171b33e880e6a6fd5cd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42581 } } peers { permanent_uuid: "5fd1996f61cc4be2858c26c0f009154b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45245 } }
I20260929 19:00:58.624244 2700 leader_election.cc:288] T 31a772fb86714054ba38e1a099c7fedf P 3411d1f14ac94407b9fe8e491584772b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0725f0c1e7c74171b33e880e6a6fd5cd (127.0.98.2:42581), 5fd1996f61cc4be2858c26c0f009154b (127.0.98.3:45245)
I20260929 19:00:58.624394 2433 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31a772fb86714054ba38e1a099c7fedf" candidate_uuid: "3411d1f14ac94407b9fe8e491584772b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0725f0c1e7c74171b33e880e6a6fd5cd" is_pre_election: true
I20260929 19:00:58.624509 2433 raft_consensus.cc:2454] T 31a772fb86714054ba38e1a099c7fedf P 0725f0c1e7c74171b33e880e6a6fd5cd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3411d1f14ac94407b9fe8e491584772b in term 0.
I20260929 19:00:58.624671 2225 leader_election.cc:302] T 31a772fb86714054ba38e1a099c7fedf P 3411d1f14ac94407b9fe8e491584772b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0725f0c1e7c74171b33e880e6a6fd5cd, 3411d1f14ac94407b9fe8e491584772b; no voters:
I20260929 19:00:58.624783 2700 raft_consensus.cc:2781] T 31a772fb86714054ba38e1a099c7fedf P 3411d1f14ac94407b9fe8e491584772b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 19:00:58.624859 2700 raft_consensus.cc:482] T 31a772fb86714054ba38e1a099c7fedf P 3411d1f14ac94407b9fe8e491584772b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 19:00:58.624794 2574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31a772fb86714054ba38e1a099c7fedf" candidate_uuid: "3411d1f14ac94407b9fe8e491584772b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fd1996f61cc4be2858c26c0f009154b" is_pre_election: true
I20260929 19:00:58.624889 2700 raft_consensus.cc:3037] T 31a772fb86714054ba38e1a099c7fedf P 3411d1f14ac94407b9fe8e491584772b [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:00:58.624907 2574 raft_consensus.cc:2454] T 31a772fb86714054ba38e1a099c7fedf P 5fd1996f61cc4be2858c26c0f009154b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3411d1f14ac94407b9fe8e491584772b in term 0.
I20260929 19:00:58.625619 2700 raft_consensus.cc:504] T 31a772fb86714054ba38e1a099c7fedf P 3411d1f14ac94407b9fe8e491584772b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3411d1f14ac94407b9fe8e491584772b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42221 } } peers { permanent_uuid: "0725f0c1e7c74171b33e880e6a6fd5cd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42581 } } peers { permanent_uuid: "5fd1996f61cc4be2858c26c0f009154b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45245 } }
I20260929 19:00:58.625756 2700 leader_election.cc:288] T 31a772fb86714054ba38e1a099c7fedf P 3411d1f14ac94407b9fe8e491584772b [CANDIDATE]: Term 1 election: Requested vote from peers 0725f0c1e7c74171b33e880e6a6fd5cd (127.0.98.2:42581), 5fd1996f61cc4be2858c26c0f009154b (127.0.98.3:45245)
I20260929 19:00:58.625902 2574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31a772fb86714054ba38e1a099c7fedf" candidate_uuid: "3411d1f14ac94407b9fe8e491584772b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5fd1996f61cc4be2858c26c0f009154b"
I20260929 19:00:58.625989 2574 raft_consensus.cc:3037] T 31a772fb86714054ba38e1a099c7fedf P 5fd1996f61cc4be2858c26c0f009154b [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:00:58.626751 2574 raft_consensus.cc:2454] T 31a772fb86714054ba38e1a099c7fedf P 5fd1996f61cc4be2858c26c0f009154b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3411d1f14ac94407b9fe8e491584772b in term 1.
I20260929 19:00:58.626883 2433 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31a772fb86714054ba38e1a099c7fedf" candidate_uuid: "3411d1f14ac94407b9fe8e491584772b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0725f0c1e7c74171b33e880e6a6fd5cd"
I20260929 19:00:58.626924 2226 leader_election.cc:302] T 31a772fb86714054ba38e1a099c7fedf P 3411d1f14ac94407b9fe8e491584772b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3411d1f14ac94407b9fe8e491584772b, 5fd1996f61cc4be2858c26c0f009154b; no voters:
I20260929 19:00:58.626955 2433 raft_consensus.cc:3037] T 31a772fb86714054ba38e1a099c7fedf P 0725f0c1e7c74171b33e880e6a6fd5cd [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:00:58.627000 2700 raft_consensus.cc:2781] T 31a772fb86714054ba38e1a099c7fedf P 3411d1f14ac94407b9fe8e491584772b [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:00:58.627045 2700 raft_consensus.cc:686] T 31a772fb86714054ba38e1a099c7fedf P 3411d1f14ac94407b9fe8e491584772b [term 1 LEADER]: Becoming Leader. State: Replica: 3411d1f14ac94407b9fe8e491584772b, State: Running, Role: LEADER
I20260929 19:00:58.627102 2700 consensus_queue.cc:237] T 31a772fb86714054ba38e1a099c7fedf P 3411d1f14ac94407b9fe8e491584772b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3411d1f14ac94407b9fe8e491584772b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42221 } } peers { permanent_uuid: "0725f0c1e7c74171b33e880e6a6fd5cd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42581 } } peers { permanent_uuid: "5fd1996f61cc4be2858c26c0f009154b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45245 } }
I20260929 19:00:58.627663 2137 catalog_manager.cc:5495] T 31a772fb86714054ba38e1a099c7fedf P 3411d1f14ac94407b9fe8e491584772b reported cstate change: term changed from 0 to 1, leader changed from <none> to 3411d1f14ac94407b9fe8e491584772b (127.0.98.1). New cstate: current_term: 1 leader_uuid: "3411d1f14ac94407b9fe8e491584772b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3411d1f14ac94407b9fe8e491584772b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42221 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0725f0c1e7c74171b33e880e6a6fd5cd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42581 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5fd1996f61cc4be2858c26c0f009154b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45245 } health_report { overall_health: UNKNOWN } } }
I20260929 19:00:58.627717 2433 raft_consensus.cc:2454] T 31a772fb86714054ba38e1a099c7fedf P 0725f0c1e7c74171b33e880e6a6fd5cd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3411d1f14ac94407b9fe8e491584772b in term 1.
I20260929 19:00:58.652010 2743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:00:58.650071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37914 (local address 127.0.98.1:42221)
0929 19:00:58.650135 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0929 19:00:58.650136 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 19:00:58.650143 (+ 7us) server_negotiation.cc:407] Connection header received
0929 19:00:58.650163 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:00:58.650163 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:00:58.650173 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:00:58.650218 (+ 45us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 19:00:58.650342 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:58.650798 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:58.651459 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:00:58.651647 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:00:58.651701 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:00:58.651819 (+ 118us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 19:00:58.651843 (+ 24us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 19:00:58.651881 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0929 19:00:58.651933 (+ 52us) server_negotiation.cc:299] Negotiation successful
0929 19:00:58.651954 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 19:00:58.654115 2433 raft_consensus.cc:1261] T 31a772fb86714054ba38e1a099c7fedf P 0725f0c1e7c74171b33e880e6a6fd5cd [term 1 FOLLOWER]: Refusing update from remote peer 3411d1f14ac94407b9fe8e491584772b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 19:00:58.654110 2574 raft_consensus.cc:1261] T 31a772fb86714054ba38e1a099c7fedf P 5fd1996f61cc4be2858c26c0f009154b [term 1 FOLLOWER]: Refusing update from remote peer 3411d1f14ac94407b9fe8e491584772b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 19:00:58.654320 2719 consensus_queue.cc:1035] T 31a772fb86714054ba38e1a099c7fedf P 3411d1f14ac94407b9fe8e491584772b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5fd1996f61cc4be2858c26c0f009154b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45245 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 19:00:58.654422 2700 consensus_queue.cc:1035] T 31a772fb86714054ba38e1a099c7fedf P 3411d1f14ac94407b9fe8e491584772b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0725f0c1e7c74171b33e880e6a6fd5cd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 19:00:58.658370 2707 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260929 19:01:00.174541 2137 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42844:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 19:01:00.174638 2137 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42844:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 19:01:00.175556 2137 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 624b000e6c514bc39d63e63ee983eca5: Sending DeleteTablet for 3 replicas of tablet 31a772fb86714054ba38e1a099c7fedf
I20260929 19:01:00.175814 2414 tablet_service.cc:1514] Processing DeleteTablet for tablet 31a772fb86714054ba38e1a099c7fedf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:01:00 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53396
I20260929 19:01:00.175849 2274 tablet_service.cc:1514] Processing DeleteTablet for tablet 31a772fb86714054ba38e1a099c7fedf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:01:00 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55212
I20260929 19:01:00.175846 2553 tablet_service.cc:1514] Processing DeleteTablet for tablet 31a772fb86714054ba38e1a099c7fedf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:01:00 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49880
I20260929 19:01:00.175941 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUbnLHE/build/release/bin/kudu with pid 2209
I20260929 19:01:00.176865 2789 tablet_replica.cc:330] T 31a772fb86714054ba38e1a099c7fedf P 5fd1996f61cc4be2858c26c0f009154b: stopping tablet replica
I20260929 19:01:00.176937 2789 raft_consensus.cc:2229] T 31a772fb86714054ba38e1a099c7fedf P 5fd1996f61cc4be2858c26c0f009154b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 19:01:00.176977 2789 raft_consensus.cc:2258] T 31a772fb86714054ba38e1a099c7fedf P 5fd1996f61cc4be2858c26c0f009154b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 19:01:00.177220 2789 ts_tablet_manager.cc:1905] T 31a772fb86714054ba38e1a099c7fedf P 5fd1996f61cc4be2858c26c0f009154b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 19:01:00.179174 2789 ts_tablet_manager.cc:1918] T 31a772fb86714054ba38e1a099c7fedf P 5fd1996f61cc4be2858c26c0f009154b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 19:01:00.179242 2789 log.cc:1196] T 31a772fb86714054ba38e1a099c7fedf P 5fd1996f61cc4be2858c26c0f009154b: Deleting WAL directory at /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708447806542-392-0/minicluster-data/ts-2/wal/wals/31a772fb86714054ba38e1a099c7fedf
I20260929 19:01:00.179466 2789 ts_tablet_manager.cc:1939] T 31a772fb86714054ba38e1a099c7fedf P 5fd1996f61cc4be2858c26c0f009154b: Deleting consensus metadata
I20260929 19:01:00.179804 2122 catalog_manager.cc:4841] TS 5fd1996f61cc4be2858c26c0f009154b (127.0.98.3:45245): tablet 31a772fb86714054ba38e1a099c7fedf (table test-table [id=38f354c0b6bb4c8fa1cc44db19d50817]) successfully deleted
I20260929 19:01:00.180912 2787 tablet_replica.cc:330] T 31a772fb86714054ba38e1a099c7fedf P 0725f0c1e7c74171b33e880e6a6fd5cd: stopping tablet replica
I20260929 19:01:00.180985 2787 raft_consensus.cc:2229] T 31a772fb86714054ba38e1a099c7fedf P 0725f0c1e7c74171b33e880e6a6fd5cd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 19:01:00.181025 2787 raft_consensus.cc:2258] T 31a772fb86714054ba38e1a099c7fedf P 0725f0c1e7c74171b33e880e6a6fd5cd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 19:01:00.181267 2787 ts_tablet_manager.cc:1905] T 31a772fb86714054ba38e1a099c7fedf P 0725f0c1e7c74171b33e880e6a6fd5cd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 19:01:00.183156 2787 ts_tablet_manager.cc:1918] T 31a772fb86714054ba38e1a099c7fedf P 0725f0c1e7c74171b33e880e6a6fd5cd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 19:01:00.183219 2787 log.cc:1196] T 31a772fb86714054ba38e1a099c7fedf P 0725f0c1e7c74171b33e880e6a6fd5cd: Deleting WAL directory at /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790708447806542-392-0/minicluster-data/ts-1/wal/wals/31a772fb86714054ba38e1a099c7fedf
W20260929 19:01:00.183430 2122 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:42221 (error 108)
I20260929 19:01:00.183650 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUbnLHE/build/release/bin/kudu with pid 2349
I20260929 19:01:00.189051 2791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:00.183787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42221 (local address 127.0.0.1:37922)
0929 19:01:00.188939 (+ 5152us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:00.189002 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":5121,"thread_start_us":57,"threads_started":1}
W20260929 19:01:00.189210 2122 catalog_manager.cc:4584] TS 3411d1f14ac94407b9fe8e491584772b (127.0.98.1:42221): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 31a772fb86714054ba38e1a099c7fedf: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42221: connect: Connection refused (error 111)
I20260929 19:01:00.189570 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUbnLHE/build/release/bin/kudu with pid 2489
I20260929 19:01:00.189615 2791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:00.189519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42581 (local address 127.0.0.1:42388)
0929 19:01:00.189557 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:00.189590 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:42581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
I20260929 19:01:00.195036 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUbnLHE/build/release/bin/kudu with pid 2105
2026-09-29T19:01:00Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (5753 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708447806542-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 19:01:00 dist-test-slave-lqkv krb5kdc[2794](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 19:01:00 dist-test-slave-lqkv krb5kdc[2794](info): set up 2 sockets
Sep 29 19:01:00 dist-test-slave-lqkv krb5kdc[2794](info): commencing operation
krb5kdc: starting...
W20260929 19:01:01.236562 392 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708447806542-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708447806542-392-0/krb5kdc/test-user.keytab.
Sep 29 19:01:01 dist-test-slave-lqkv krb5kdc[2794](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708461, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T19:01:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T19:01:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708447806542-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708447806542-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 19:01:01.349315 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUbnLHE/build/release/bin/kudu
/tmp/dist-test-taskUbnLHE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708447806542-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708447806542-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708447806542-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708447806542-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44917
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708447806542-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44917
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708447806542-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708447806542-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708447806542-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:01:01.427582 2807 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:01:01.427765 2807 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:01:01.427820 2807 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:01:01.429222 2807 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 19:01:01.429266 2807 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:01:01.429280 2807 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:01:01.429293 2807 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 19:01:01.429306 2807 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 19:01:01.430688 2807 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708447806542-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708447806542-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44917
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708447806542-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44917
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708447806542-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2807
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708447806542-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 18:09:13 UTC on ccycloud.quasar-hpkalf.root.comops.site
build id 4680913
I20260929 19:01:01.430866 2807 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:01:01.431025 2807 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:01:01.435536 2813 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:01:01.435544 2815 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:01:01.435544 2812 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:01:01.435599 2807 server_base.cc:1034] running on GCE node
I20260929 19:01:01.435956 2807 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:01:01.436233 2807 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:01:01.437393 2807 hybrid_clock.cc:648] HybridClock initialized: now 1790708461437388 us; error 49 us; skew 500 ppm
Sep 29 19:01:01 dist-test-slave-lqkv krb5kdc[2794](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708461, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:01:01.439710 2807 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 19:01:01.440151 2807 webserver.cc:460] Webserver started at http://127.0.98.62:44087/ using document root <none> and password file <none>
I20260929 19:01:01.440398 2807 fs_manager.cc:359] Metadata directory not provided
I20260929 19:01:01.440461 2807 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:01:01.440596 2807 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:01:01.441658 2807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708447806542-392-0/minicluster-data/master-0/data/instance:
uuid: "fafa9447477b4cd2be2c8b89ad11284d"
format_stamp: "Formatted at 2026-09-29 19:01:01 on dist-test-slave-lqkv"
server_key: "458056f046c811832bfd07d04bb5fa85"
server_key_iv: "62748d67e79e42912d2f4c020e860faa"
server_key_version: "encryptionkey@0"
I20260929 19:01:01.442001 2807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708447806542-392-0/minicluster-data/master-0/wal/instance:
uuid: "fafa9447477b4cd2be2c8b89ad11284d"
format_stamp: "Formatted at 2026-09-29 19:01:01 on dist-test-slave-lqkv"
server_key: "458056f046c811832bfd07d04bb5fa85"
server_key_iv: "62748d67e79e42912d2f4c020e860faa"
server_key_version: "encryptionkey@0"
I20260929 19:01:01.443702 2807 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20260929 19:01:01.444468 2822 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:01.444646 2807 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 19:01:01.444747 2807 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708447806542-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708447806542-392-0/minicluster-data/master-0/wal
uuid: "fafa9447477b4cd2be2c8b89ad11284d"
format_stamp: "Formatted at 2026-09-29 19:01:01 on dist-test-slave-lqkv"
server_key: "458056f046c811832bfd07d04bb5fa85"
server_key_iv: "62748d67e79e42912d2f4c020e860faa"
server_key_version: "encryptionkey@0"
I20260929 19:01:01.444849 2807 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708447806542-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708447806542-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708447806542-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:01:01.457733 2807 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:01:01.459501 2807 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:01:01.459653 2807 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:01:01.463748 2807 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44917
I20260929 19:01:01.463778 2874 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44917 every 8 connection(s)
I20260929 19:01:01.464257 2807 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708447806542-392-0/minicluster-data/master-0/data/info.pb
I20260929 19:01:01.464936 2875 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:01:01.467170 2875 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fafa9447477b4cd2be2c8b89ad11284d: Bootstrap starting.
I20260929 19:01:01.467737 2875 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fafa9447477b4cd2be2c8b89ad11284d: Neither blocks nor log segments found. Creating new log.
I20260929 19:01:01.468029 2875 log.cc:824] T 00000000000000000000000000000000 P fafa9447477b4cd2be2c8b89ad11284d: Log is configured to *not* fsync() on all Append() calls
I20260929 19:01:01.468705 2875 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fafa9447477b4cd2be2c8b89ad11284d: No bootstrap required, opened a new log
I20260929 19:01:01.470266 2875 raft_consensus.cc:348] T 00000000000000000000000000000000 P fafa9447477b4cd2be2c8b89ad11284d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fafa9447477b4cd2be2c8b89ad11284d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44917 } }
I20260929 19:01:01.470398 2875 raft_consensus.cc:374] T 00000000000000000000000000000000 P fafa9447477b4cd2be2c8b89ad11284d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:01:01.470419 2875 raft_consensus.cc:729] T 00000000000000000000000000000000 P fafa9447477b4cd2be2c8b89ad11284d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fafa9447477b4cd2be2c8b89ad11284d, State: Initialized, Role: FOLLOWER
I20260929 19:01:01.470587 2875 consensus_queue.cc:260] T 00000000000000000000000000000000 P fafa9447477b4cd2be2c8b89ad11284d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fafa9447477b4cd2be2c8b89ad11284d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44917 } }
I20260929 19:01:01.470650 2875 raft_consensus.cc:388] T 00000000000000000000000000000000 P fafa9447477b4cd2be2c8b89ad11284d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 19:01:01.470671 2875 raft_consensus.cc:482] T 00000000000000000000000000000000 P fafa9447477b4cd2be2c8b89ad11284d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 19:01:01.470698 2875 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fafa9447477b4cd2be2c8b89ad11284d [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:01:01.471572 2875 raft_consensus.cc:504] T 00000000000000000000000000000000 P fafa9447477b4cd2be2c8b89ad11284d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fafa9447477b4cd2be2c8b89ad11284d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44917 } }
I20260929 19:01:01.471660 2875 leader_election.cc:302] T 00000000000000000000000000000000 P fafa9447477b4cd2be2c8b89ad11284d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fafa9447477b4cd2be2c8b89ad11284d; no voters:
I20260929 19:01:01.471822 2875 leader_election.cc:288] T 00000000000000000000000000000000 P fafa9447477b4cd2be2c8b89ad11284d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 19:01:01.471905 2878 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fafa9447477b4cd2be2c8b89ad11284d [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:01:01.472076 2878 raft_consensus.cc:686] T 00000000000000000000000000000000 P fafa9447477b4cd2be2c8b89ad11284d [term 1 LEADER]: Becoming Leader. State: Replica: fafa9447477b4cd2be2c8b89ad11284d, State: Running, Role: LEADER
I20260929 19:01:01.472110 2875 sys_catalog.cc:564] T 00000000000000000000000000000000 P fafa9447477b4cd2be2c8b89ad11284d [sys.catalog]: configured and running, proceeding with master startup.
I20260929 19:01:01.472187 2878 consensus_queue.cc:237] T 00000000000000000000000000000000 P fafa9447477b4cd2be2c8b89ad11284d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fafa9447477b4cd2be2c8b89ad11284d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44917 } }
I20260929 19:01:01.472590 2880 sys_catalog.cc:455] T 00000000000000000000000000000000 P fafa9447477b4cd2be2c8b89ad11284d [sys.catalog]: SysCatalogTable state changed. Reason: New leader fafa9447477b4cd2be2c8b89ad11284d. Latest consensus state: current_term: 1 leader_uuid: "fafa9447477b4cd2be2c8b89ad11284d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fafa9447477b4cd2be2c8b89ad11284d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44917 } } }
I20260929 19:01:01.472707 2880 sys_catalog.cc:458] T 00000000000000000000000000000000 P fafa9447477b4cd2be2c8b89ad11284d [sys.catalog]: This master's current role is: LEADER
I20260929 19:01:01.472575 2879 sys_catalog.cc:455] T 00000000000000000000000000000000 P fafa9447477b4cd2be2c8b89ad11284d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fafa9447477b4cd2be2c8b89ad11284d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fafa9447477b4cd2be2c8b89ad11284d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44917 } } }
I20260929 19:01:01.472961 2879 sys_catalog.cc:458] T 00000000000000000000000000000000 P fafa9447477b4cd2be2c8b89ad11284d [sys.catalog]: This master's current role is: LEADER
I20260929 19:01:01.473248 2887 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 19:01:01.473812 2887 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 19:01:01.473866 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUbnLHE/build/release/bin/kudu as pid 2807
I20260929 19:01:01.473955 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708447806542-392-0/minicluster-data/master-0/wal/instance
I20260929 19:01:01.474170 392 external_mini_cluster.cc:1428] Setting key 6faa7cda6ce23ba901d72dfa619fd0af
I20260929 19:01:01.475379 2887 catalog_manager.cc:1347] Generated new cluster ID: fab9f2cd0b574057887c96b2ac9cd0e4
I20260929 19:01:01.475427 2887 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 29 19:01:01 dist-test-slave-lqkv krb5kdc[2794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708461, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 19:01:01.485601 2897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:01.474836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52708 (local address 127.0.98.62:44917)
0929 19:01:01.475007 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:01.475016 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:01.475047 (+ 31us) server_negotiation.cc:407] Connection header received
0929 19:01:01.475161 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:01.475167 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:01.475247 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:01.475417 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:01:01.476155 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:01.477585 (+ 1430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:01.478642 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:01.479247 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:01.479338 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:01.482134 (+ 2796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:01:01.482139 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:01:01.482139 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:01:01.482151 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:01:01.484040 (+ 1889us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:01.484399 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:01.484401 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:01.484402 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:01.484503 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:01.484690 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:01.484690 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:01.484691 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:01.484997 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:01:01.485170 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0929 19:01:01.485249 (+ 79us) server_negotiation.cc:299] Negotiation successful
0929 19:01:01.485380 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":76,"threads_started":1}
I20260929 19:01:01.511282 2887 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 19:01:01.512020 2887 catalog_manager.cc:1504] Loading token signing keys...
I20260929 19:01:01.519927 2887 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fafa9447477b4cd2be2c8b89ad11284d: Generated new TSK 0
I20260929 19:01:01.520134 2887 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708447806542-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708447806542-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260929 19:01:01.545274 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUbnLHE/build/release/bin/kudu
/tmp/dist-test-taskUbnLHE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708447806542-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708447806542-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708447806542-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708447806542-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44917
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708447806542-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708447806542-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708447806542-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708447806542-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:01:01.624384 2901 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:01:01.624567 2901 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:01:01.624590 2901 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:01:01.625651 2901 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:01:01.625685 2901 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:01:01.625710 2901 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 19:01:01.627138 2901 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708447806542-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708447806542-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708447806542-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708447806542-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44917
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2901
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708447806542-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 18:09:13 UTC on ccycloud.quasar-hpkalf.root.comops.site
build id 4680913
I20260929 19:01:01.627308 2901 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:01:01.627473 2901 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:01:01.632208 2909 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:01:01.632242 2907 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:01:01.632222 2906 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:01:01.632471 2901 server_base.cc:1034] running on GCE node
I20260929 19:01:01.632664 2901 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:01:01.632910 2901 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:01:01.634080 2901 hybrid_clock.cc:648] HybridClock initialized: now 1790708461634049 us; error 48 us; skew 500 ppm
Sep 29 19:01:01 dist-test-slave-lqkv krb5kdc[2794](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708461, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:01:01.636575 2901 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 19:01:01.637027 2901 webserver.cc:460] Webserver started at http://127.0.98.1:43221/ using document root <none> and password file <none>
I20260929 19:01:01.637264 2901 fs_manager.cc:359] Metadata directory not provided
I20260929 19:01:01.637311 2901 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:01:01.637434 2901 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:01:01.638401 2901 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708447806542-392-0/minicluster-data/ts-0/data/instance:
uuid: "c5dd8456a5ba48f09764498c11906111"
format_stamp: "Formatted at 2026-09-29 19:01:01 on dist-test-slave-lqkv"
server_key: "e59a61e230a1030258a4aa8f925ba12d"
server_key_iv: "9d1407d1c9dd9dbc2d22699e100f4920"
server_key_version: "encryptionkey@0"
I20260929 19:01:01.638760 2901 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708447806542-392-0/minicluster-data/ts-0/wal/instance:
uuid: "c5dd8456a5ba48f09764498c11906111"
format_stamp: "Formatted at 2026-09-29 19:01:01 on dist-test-slave-lqkv"
server_key: "e59a61e230a1030258a4aa8f925ba12d"
server_key_iv: "9d1407d1c9dd9dbc2d22699e100f4920"
server_key_version: "encryptionkey@0"
I20260929 19:01:01.639989 2901 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 19:01:01.640771 2916 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:01.640988 2901 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 19:01:01.641070 2901 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708447806542-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708447806542-392-0/minicluster-data/ts-0/wal
uuid: "c5dd8456a5ba48f09764498c11906111"
format_stamp: "Formatted at 2026-09-29 19:01:01 on dist-test-slave-lqkv"
server_key: "e59a61e230a1030258a4aa8f925ba12d"
server_key_iv: "9d1407d1c9dd9dbc2d22699e100f4920"
server_key_version: "encryptionkey@0"
I20260929 19:01:01.641130 2901 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708447806542-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708447806542-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708447806542-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:01:01.656279 2901 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:01:01.658201 2901 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:01:01.658308 2901 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:01:01.658473 2901 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:01:01.658712 2901 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:01:01.658742 2901 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:01.658764 2901 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:01:01.658776 2901 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:01.665776 2901 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43307
I20260929 19:01:01.665858 3029 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43307 every 8 connection(s)
I20260929 19:01:01.666188 2901 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708447806542-392-0/minicluster-data/ts-0/data/info.pb
I20260929 19:01:01.669797 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUbnLHE/build/release/bin/kudu as pid 2901
I20260929 19:01:01.669883 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708447806542-392-0/minicluster-data/ts-0/wal/instance
I20260929 19:01:01.670039 392 external_mini_cluster.cc:1428] Setting key cfb04bc81a8b2928728e80a5b8718b07
Sep 29 19:01:01 dist-test-slave-lqkv krb5kdc[2794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708461, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 19:01:01.677489 2897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:01.666887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58741 (local address 127.0.98.62:44917)
0929 19:01:01.666946 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:01.666948 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:01.667154 (+ 206us) server_negotiation.cc:407] Connection header received
0929 19:01:01.668348 (+ 1194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:01.668350 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:01.668362 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:01.668411 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:01:01.669349 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:01.670157 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:01.671573 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:01.671789 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:01.671850 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:01.674414 (+ 2564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:01:01.674415 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:01:01.674416 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:01:01.674424 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:01:01.675925 (+ 1501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:01.676444 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:01.676445 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:01.676445 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:01.676494 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:01.676769 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:01.676770 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:01.676770 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:01.676929 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:01:01.676980 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0929 19:01:01.677349 (+ 369us) server_negotiation.cc:299] Negotiation successful
0929 19:01:01.677424 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260929 19:01:01.677726 3032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:01.666865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44917 (local address 127.0.98.1:58741)
0929 19:01:01.667020 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:01.667075 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0929 19:01:01.668219 (+ 1144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:01:01.668526 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:01:01.668530 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:01:01.668930 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:01:01.669258 (+ 328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:01.669265 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:01.670273 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:01.670274 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:01.671474 (+ 1200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:01.671477 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:01.671878 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:01.671879 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:01.672052 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:01.672700 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:01:01.672709 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:01:01.674315 (+ 1606us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:01:01.676010 (+ 1695us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:01:01.676018 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:01:01.676020 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:01:01.676360 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:01:01.676554 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:01:01.676555 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:01:01.676555 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:01:01.676701 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:01:01.676995 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:01:01.677001 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:01:01.677179 (+ 178us) client_negotiation.cc:769] Sending connection context
0929 19:01:01.677303 (+ 124us) client_negotiation.cc:240] Negotiation successful
0929 19:01:01.677479 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":84,"threads_started":1}
I20260929 19:01:01.678259 3030 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44917
I20260929 19:01:01.678369 3030 heartbeater.cc:464] Registering TS with master...
I20260929 19:01:01.678558 3030 heartbeater.cc:511] Master 127.0.98.62:44917 requested a full tablet report, sending...
I20260929 19:01:01.679085 2839 ts_manager.cc:194] Registered new tserver with Master: c5dd8456a5ba48f09764498c11906111 (127.0.98.1:43307)
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260929 19:01:01.679745 2839 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:58741
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708447806542-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708447806542-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260929 19:01:01.695124 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUbnLHE/build/release/bin/kudu
/tmp/dist-test-taskUbnLHE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708447806542-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708447806542-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708447806542-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708447806542-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44917
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708447806542-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708447806542-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708447806542-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708447806542-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:01:01.774015 3035 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:01:01.774241 3035 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:01:01.774266 3035 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:01:01.775393 3035 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:01:01.775434 3035 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:01:01.775486 3035 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 19:01:01.776998 3035 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708447806542-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708447806542-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708447806542-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708447806542-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44917
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3035
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708447806542-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 18:09:13 UTC on ccycloud.quasar-hpkalf.root.comops.site
build id 4680913
I20260929 19:01:01.777240 3035 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:01:01.777465 3035 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:01:01.782187 3040 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:01:01.782184 3043 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:01:01.782357 3035 server_base.cc:1034] running on GCE node
W20260929 19:01:01.782251 3041 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:01:01.782644 3035 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:01:01.782840 3035 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:01:01.783972 3035 hybrid_clock.cc:648] HybridClock initialized: now 1790708461783958 us; error 27 us; skew 500 ppm
Sep 29 19:01:01 dist-test-slave-lqkv krb5kdc[2794](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708461, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:01:01.786413 3035 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 19:01:01.786823 3035 webserver.cc:460] Webserver started at http://127.0.98.2:43827/ using document root <none> and password file <none>
I20260929 19:01:01.787030 3035 fs_manager.cc:359] Metadata directory not provided
I20260929 19:01:01.787086 3035 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:01:01.787258 3035 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:01:01.788270 3035 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708447806542-392-0/minicluster-data/ts-1/data/instance:
uuid: "fd6a3aaca96a435e9c464aa811a0112a"
format_stamp: "Formatted at 2026-09-29 19:01:01 on dist-test-slave-lqkv"
server_key: "3fa476dcda5b5b1aab73f54158f6e555"
server_key_iv: "4f526d99d5d599ec0ebf9a5c63683fb7"
server_key_version: "encryptionkey@0"
I20260929 19:01:01.788625 3035 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708447806542-392-0/minicluster-data/ts-1/wal/instance:
uuid: "fd6a3aaca96a435e9c464aa811a0112a"
format_stamp: "Formatted at 2026-09-29 19:01:01 on dist-test-slave-lqkv"
server_key: "3fa476dcda5b5b1aab73f54158f6e555"
server_key_iv: "4f526d99d5d599ec0ebf9a5c63683fb7"
server_key_version: "encryptionkey@0"
I20260929 19:01:01.789927 3035 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260929 19:01:01.790650 3050 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:01.790830 3035 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 19:01:01.790918 3035 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708447806542-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708447806542-392-0/minicluster-data/ts-1/wal
uuid: "fd6a3aaca96a435e9c464aa811a0112a"
format_stamp: "Formatted at 2026-09-29 19:01:01 on dist-test-slave-lqkv"
server_key: "3fa476dcda5b5b1aab73f54158f6e555"
server_key_iv: "4f526d99d5d599ec0ebf9a5c63683fb7"
server_key_version: "encryptionkey@0"
I20260929 19:01:01.790992 3035 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708447806542-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708447806542-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708447806542-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:01:01.802971 3035 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:01:01.805241 3035 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:01:01.805401 3035 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:01:01.805631 3035 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:01:01.805923 3035 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:01:01.805955 3035 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:01.805977 3035 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:01:01.805989 3035 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:01.812338 3035 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37665
I20260929 19:01:01.812414 3163 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37665 every 8 connection(s)
I20260929 19:01:01.812700 3035 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708447806542-392-0/minicluster-data/ts-1/data/info.pb
Sep 29 19:01:01 dist-test-slave-lqkv krb5kdc[2794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708461, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 19:01:01.819272 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUbnLHE/build/release/bin/kudu as pid 3035
I20260929 19:01:01.819375 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708447806542-392-0/minicluster-data/ts-1/wal/instance
I20260929 19:01:01.819563 392 external_mini_cluster.cc:1428] Setting key 158e5cf6f07171308159df6b72dccf7f
I20260929 19:01:01.822224 2897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:01.813469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55833 (local address 127.0.98.62:44917)
0929 19:01:01.813533 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:01.813535 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:01.813730 (+ 195us) server_negotiation.cc:407] Connection header received
0929 19:01:01.814620 (+ 890us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:01.814621 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:01.814631 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:01.814692 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:01:01.815283 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:01.815768 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:01.816719 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:01.816947 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:01.817004 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:01.818938 (+ 1934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:01:01.818939 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:01:01.818939 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:01:01.818946 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:01:01.820773 (+ 1827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:01.821214 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:01.821215 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:01.821216 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:01.821271 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:01.821570 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:01.821571 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:01.821572 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:01.821721 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:01:01.821780 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0929 19:01:01.822098 (+ 318us) server_negotiation.cc:299] Negotiation successful
0929 19:01:01.822162 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260929 19:01:01.822522 3166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:01.813428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44917 (local address 127.0.98.2:55833)
0929 19:01:01.813585 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:01.813625 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 19:01:01.814512 (+ 887us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:01:01.814759 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:01:01.814762 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:01:01.814997 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:01:01.815201 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:01.815205 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:01.815842 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:01.815842 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:01.816640 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:01.816642 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:01.817023 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:01.817024 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:01.817128 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:01.817586 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:01:01.817593 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:01:01.818850 (+ 1257us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:01:01.820891 (+ 2041us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:01:01.820897 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:01:01.820899 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:01:01.821117 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:01:01.821344 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:01:01.821344 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:01:01.821345 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:01:01.821478 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:01:01.821792 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:01:01.821797 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:01:01.821959 (+ 162us) client_negotiation.cc:769] Sending connection context
0929 19:01:01.822076 (+ 117us) client_negotiation.cc:240] Negotiation successful
0929 19:01:01.822265 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":84,"threads_started":1}
I20260929 19:01:01.822959 3164 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44917
I20260929 19:01:01.823089 3164 heartbeater.cc:464] Registering TS with master...
I20260929 19:01:01.823279 3164 heartbeater.cc:511] Master 127.0.98.62:44917 requested a full tablet report, sending...
I20260929 19:01:01.823642 2839 ts_manager.cc:194] Registered new tserver with Master: fd6a3aaca96a435e9c464aa811a0112a (127.0.98.2:37665)
I20260929 19:01:01.824174 2839 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:55833
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708447806542-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708447806542-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260929 19:01:01.843632 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUbnLHE/build/release/bin/kudu
/tmp/dist-test-taskUbnLHE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708447806542-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708447806542-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708447806542-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708447806542-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44917
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708447806542-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708447806542-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708447806542-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708447806542-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:01:01.922503 3169 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:01:01.922752 3169 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:01:01.922789 3169 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:01:01.923921 3169 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:01:01.923985 3169 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:01:01.924031 3169 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 19:01:01.925554 3169 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708447806542-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708447806542-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708447806542-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708447806542-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44917
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3169
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708447806542-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 18:09:13 UTC on ccycloud.quasar-hpkalf.root.comops.site
build id 4680913
I20260929 19:01:01.925822 3169 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:01:01.926040 3169 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:01:01.930841 3175 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:01:01.930984 3169 server_base.cc:1034] running on GCE node
W20260929 19:01:01.930848 3174 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:01:01.930848 3177 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:01:01.931330 3169 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:01:01.931602 3169 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:01:01.932762 3169 hybrid_clock.cc:648] HybridClock initialized: now 1790708461932748 us; error 26 us; skew 500 ppm
Sep 29 19:01:01 dist-test-slave-lqkv krb5kdc[2794](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708461, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:01:01.935290 3169 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 19:01:01.935732 3169 webserver.cc:460] Webserver started at http://127.0.98.3:36605/ using document root <none> and password file <none>
I20260929 19:01:01.935953 3169 fs_manager.cc:359] Metadata directory not provided
I20260929 19:01:01.936021 3169 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:01:01.936167 3169 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:01:01.937227 3169 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708447806542-392-0/minicluster-data/ts-2/data/instance:
uuid: "fbe6637d48c94468918b11776c0cbed1"
format_stamp: "Formatted at 2026-09-29 19:01:01 on dist-test-slave-lqkv"
server_key: "f4b434bfa5e147c94cbb53a1934e16cb"
server_key_iv: "c7265124b8cc2365b7902ccc2c431d25"
server_key_version: "encryptionkey@0"
I20260929 19:01:01.937616 3169 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708447806542-392-0/minicluster-data/ts-2/wal/instance:
uuid: "fbe6637d48c94468918b11776c0cbed1"
format_stamp: "Formatted at 2026-09-29 19:01:01 on dist-test-slave-lqkv"
server_key: "f4b434bfa5e147c94cbb53a1934e16cb"
server_key_iv: "c7265124b8cc2365b7902ccc2c431d25"
server_key_version: "encryptionkey@0"
I20260929 19:01:01.938863 3169 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 19:01:01.939566 3184 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:01.939733 3169 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 19:01:01.939824 3169 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708447806542-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708447806542-392-0/minicluster-data/ts-2/wal
uuid: "fbe6637d48c94468918b11776c0cbed1"
format_stamp: "Formatted at 2026-09-29 19:01:01 on dist-test-slave-lqkv"
server_key: "f4b434bfa5e147c94cbb53a1934e16cb"
server_key_iv: "c7265124b8cc2365b7902ccc2c431d25"
server_key_version: "encryptionkey@0"
I20260929 19:01:01.939901 3169 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708447806542-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708447806542-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708447806542-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:01:01.976289 3169 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:01:01.978653 3169 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:01:01.978821 3169 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:01:01.979056 3169 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:01:01.979374 3169 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:01:01.979431 3169 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:01.979465 3169 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:01:01.979496 3169 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:01.985378 3169 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39131
I20260929 19:01:01.985430 3297 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39131 every 8 connection(s)
I20260929 19:01:01.985832 3169 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708447806542-392-0/minicluster-data/ts-2/data/info.pb
I20260929 19:01:01.988270 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUbnLHE/build/release/bin/kudu as pid 3169
I20260929 19:01:01.988360 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790708447806542-392-0/minicluster-data/ts-2/wal/instance
I20260929 19:01:01.988520 392 external_mini_cluster.cc:1428] Setting key de9e1e958fcb6de36691798bb9643ce1
Sep 29 19:01:01 dist-test-slave-lqkv krb5kdc[2794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708461, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 19:01:01.995424 2897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:01.986627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42221 (local address 127.0.98.62:44917)
0929 19:01:01.986698 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:01.986701 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:01.986804 (+ 103us) server_negotiation.cc:407] Connection header received
0929 19:01:01.987781 (+ 977us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:01.987782 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:01.987793 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:01.987861 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:01:01.988648 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:01.989403 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:01.990442 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:01.990626 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:01.990690 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:01.992733 (+ 2043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:01:01.992734 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:01:01.992735 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:01:01.992742 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:01:01.994227 (+ 1485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:01.994580 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:01.994581 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:01.994581 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:01.994631 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:01.994897 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:01.994898 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:01.994898 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:01.994999 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:01:01.995074 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 19:01:01.995302 (+ 228us) server_negotiation.cc:299] Negotiation successful
0929 19:01:01.995360 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260929 19:01:01.995626 3300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:01.986526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44917 (local address 127.0.98.3:42221)
0929 19:01:01.986679 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:01.986721 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 19:01:01.987655 (+ 934us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:01:01.987921 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:01:01.987925 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:01:01.988266 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:01:01.988557 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:01.988564 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:01.989491 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:01.989492 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:01.990306 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:01.990307 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:01.990709 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:01.990709 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:01.990812 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:01.991288 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:01:01.991296 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:01:01.992630 (+ 1334us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:01:01.994296 (+ 1666us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:01:01.994301 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:01:01.994303 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:01:01.994511 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:01:01.994691 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:01:01.994692 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:01:01.994692 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:01:01.994853 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:01:01.995056 (+ 203us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:01:01.995061 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:01:01.995190 (+ 129us) client_negotiation.cc:769] Sending connection context
0929 19:01:01.995272 (+ 82us) client_negotiation.cc:240] Negotiation successful
0929 19:01:01.995384 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":87,"threads_started":1}
I20260929 19:01:01.996162 3298 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44917
I20260929 19:01:01.996274 3298 heartbeater.cc:464] Registering TS with master...
I20260929 19:01:01.996462 3298 heartbeater.cc:511] Master 127.0.98.62:44917 requested a full tablet report, sending...
I20260929 19:01:01.996850 2839 ts_manager.cc:194] Registered new tserver with Master: fbe6637d48c94468918b11776c0cbed1 (127.0.98.3:39131)
I20260929 19:01:01.997239 2839 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:42221
I20260929 19:01:02.000224 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260929 19:01:02.010787 2897 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:01:02.009909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52718 (local address 127.0.98.62:44917)
0929 19:01:02.010000 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:02.010002 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:02.010032 (+ 30us) server_negotiation.cc:407] Connection header received
0929 19:01:02.010103 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:02.010104 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:02.010114 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:02.010165 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:01:02.010702 (+ 537us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52718: BlockingRecv error: recv got EOF from 127.0.0.1:52718 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 19:01:02.011569 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUbnLHE/build/release/bin/kudu with pid 2901
I20260929 19:01:02.015847 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUbnLHE/build/release/bin/kudu with pid 3035
I20260929 19:01:02.020467 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUbnLHE/build/release/bin/kudu with pid 3169
I20260929 19:01:02.024329 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUbnLHE/build/release/bin/kudu with pid 2807
2026-09-29T19:01:02Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1828 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790708447806542-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 19:01:02 dist-test-slave-lqkv krb5kdc[3310](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 19:01:02 dist-test-slave-lqkv krb5kdc[3310](info): set up 2 sockets
Sep 29 19:01:02 dist-test-slave-lqkv krb5kdc[3310](info): commencing operation
krb5kdc: starting...
W20260929 19:01:03.060539 392 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790708447806542-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790708447806542-392-0/krb5kdc/test-user.keytab.
Sep 29 19:01:03 dist-test-slave-lqkv krb5kdc[3310](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708463, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T19:01:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T19:01:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790708447806542-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790708447806542-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 19:01:03.169682 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUbnLHE/build/release/bin/kudu
/tmp/dist-test-taskUbnLHE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790708447806542-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790708447806542-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790708447806542-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790708447806542-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42395
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790708447806542-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42395
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790708447806542-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790708447806542-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790708447806542-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:01:03.247179 3323 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:01:03.247359 3323 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:01:03.247378 3323 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:01:03.248420 3323 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 19:01:03.248452 3323 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:01:03.248466 3323 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:01:03.248478 3323 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 19:01:03.248489 3323 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 19:01:03.249867 3323 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790708447806542-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790708447806542-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42395
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790708447806542-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42395
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790708447806542-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3323
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790708447806542-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 18:09:13 UTC on ccycloud.quasar-hpkalf.root.comops.site
build id 4680913
I20260929 19:01:03.250036 3323 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:01:03.250180 3323 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:01:03.254709 3331 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:01:03.254720 3329 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:01:03.254720 3328 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:01:03.254779 3323 server_base.cc:1034] running on GCE node
I20260929 19:01:03.255149 3323 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:01:03.255451 3323 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:01:03.256593 3323 hybrid_clock.cc:648] HybridClock initialized: now 1790708463256575 us; error 39 us; skew 500 ppm
Sep 29 19:01:03 dist-test-slave-lqkv krb5kdc[3310](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708463, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:01:03.258865 3323 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 19:01:03.259294 3323 webserver.cc:460] Webserver started at http://127.0.98.62:35707/ using document root <none> and password file <none>
I20260929 19:01:03.259502 3323 fs_manager.cc:359] Metadata directory not provided
I20260929 19:01:03.259574 3323 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:01:03.259703 3323 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:01:03.260699 3323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790708447806542-392-0/minicluster-data/master-0/data/instance:
uuid: "a91cf74c5953459eb403cc3461b3784a"
format_stamp: "Formatted at 2026-09-29 19:01:03 on dist-test-slave-lqkv"
server_key: "d2502e02b20c41f327b3e044ba64123b"
server_key_iv: "30dc91d58259b39e747870d11eea6890"
server_key_version: "encryptionkey@0"
I20260929 19:01:03.261075 3323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790708447806542-392-0/minicluster-data/master-0/wal/instance:
uuid: "a91cf74c5953459eb403cc3461b3784a"
format_stamp: "Formatted at 2026-09-29 19:01:03 on dist-test-slave-lqkv"
server_key: "d2502e02b20c41f327b3e044ba64123b"
server_key_iv: "30dc91d58259b39e747870d11eea6890"
server_key_version: "encryptionkey@0"
I20260929 19:01:03.262301 3323 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 19:01:03.262962 3338 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:03.263120 3323 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 19:01:03.263212 3323 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790708447806542-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790708447806542-392-0/minicluster-data/master-0/wal
uuid: "a91cf74c5953459eb403cc3461b3784a"
format_stamp: "Formatted at 2026-09-29 19:01:03 on dist-test-slave-lqkv"
server_key: "d2502e02b20c41f327b3e044ba64123b"
server_key_iv: "30dc91d58259b39e747870d11eea6890"
server_key_version: "encryptionkey@0"
I20260929 19:01:03.263295 3323 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790708447806542-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790708447806542-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790708447806542-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:01:03.274120 3323 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:01:03.275745 3323 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:01:03.275897 3323 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:01:03.279690 3323 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42395
I20260929 19:01:03.279721 3390 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42395 every 8 connection(s)
I20260929 19:01:03.280027 3323 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790708447806542-392-0/minicluster-data/master-0/data/info.pb
I20260929 19:01:03.280783 3391 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:01:03.282979 3391 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a91cf74c5953459eb403cc3461b3784a: Bootstrap starting.
I20260929 19:01:03.283609 3391 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a91cf74c5953459eb403cc3461b3784a: Neither blocks nor log segments found. Creating new log.
I20260929 19:01:03.283661 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUbnLHE/build/release/bin/kudu as pid 3323
I20260929 19:01:03.283747 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790708447806542-392-0/minicluster-data/master-0/wal/instance
I20260929 19:01:03.283954 392 external_mini_cluster.cc:1428] Setting key f87a042898266bd90d99ca6e904e3811
I20260929 19:01:03.283950 3391 log.cc:824] T 00000000000000000000000000000000 P a91cf74c5953459eb403cc3461b3784a: Log is configured to *not* fsync() on all Append() calls
I20260929 19:01:03.284621 3391 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a91cf74c5953459eb403cc3461b3784a: No bootstrap required, opened a new log
I20260929 19:01:03.286634 3391 raft_consensus.cc:348] T 00000000000000000000000000000000 P a91cf74c5953459eb403cc3461b3784a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a91cf74c5953459eb403cc3461b3784a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42395 } }
I20260929 19:01:03.286782 3391 raft_consensus.cc:374] T 00000000000000000000000000000000 P a91cf74c5953459eb403cc3461b3784a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:01:03.286827 3391 raft_consensus.cc:729] T 00000000000000000000000000000000 P a91cf74c5953459eb403cc3461b3784a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a91cf74c5953459eb403cc3461b3784a, State: Initialized, Role: FOLLOWER
I20260929 19:01:03.286995 3391 consensus_queue.cc:260] T 00000000000000000000000000000000 P a91cf74c5953459eb403cc3461b3784a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a91cf74c5953459eb403cc3461b3784a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42395 } }
I20260929 19:01:03.287081 3391 raft_consensus.cc:388] T 00000000000000000000000000000000 P a91cf74c5953459eb403cc3461b3784a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 19:01:03.287112 3391 raft_consensus.cc:482] T 00000000000000000000000000000000 P a91cf74c5953459eb403cc3461b3784a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 19:01:03.287156 3391 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a91cf74c5953459eb403cc3461b3784a [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:01:03.287784 3391 raft_consensus.cc:504] T 00000000000000000000000000000000 P a91cf74c5953459eb403cc3461b3784a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a91cf74c5953459eb403cc3461b3784a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42395 } }
I20260929 19:01:03.287900 3391 leader_election.cc:302] T 00000000000000000000000000000000 P a91cf74c5953459eb403cc3461b3784a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a91cf74c5953459eb403cc3461b3784a; no voters:
I20260929 19:01:03.288056 3391 leader_election.cc:288] T 00000000000000000000000000000000 P a91cf74c5953459eb403cc3461b3784a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 19:01:03.288081 3396 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a91cf74c5953459eb403cc3461b3784a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:01:03.288259 3396 raft_consensus.cc:686] T 00000000000000000000000000000000 P a91cf74c5953459eb403cc3461b3784a [term 1 LEADER]: Becoming Leader. State: Replica: a91cf74c5953459eb403cc3461b3784a, State: Running, Role: LEADER
I20260929 19:01:03.288298 3391 sys_catalog.cc:564] T 00000000000000000000000000000000 P a91cf74c5953459eb403cc3461b3784a [sys.catalog]: configured and running, proceeding with master startup.
I20260929 19:01:03.288408 3396 consensus_queue.cc:237] T 00000000000000000000000000000000 P a91cf74c5953459eb403cc3461b3784a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a91cf74c5953459eb403cc3461b3784a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42395 } }
I20260929 19:01:03.288844 3397 sys_catalog.cc:455] T 00000000000000000000000000000000 P a91cf74c5953459eb403cc3461b3784a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a91cf74c5953459eb403cc3461b3784a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a91cf74c5953459eb403cc3461b3784a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42395 } } }
I20260929 19:01:03.288924 3397 sys_catalog.cc:458] T 00000000000000000000000000000000 P a91cf74c5953459eb403cc3461b3784a [sys.catalog]: This master's current role is: LEADER
I20260929 19:01:03.289109 3398 sys_catalog.cc:455] T 00000000000000000000000000000000 P a91cf74c5953459eb403cc3461b3784a [sys.catalog]: SysCatalogTable state changed. Reason: New leader a91cf74c5953459eb403cc3461b3784a. Latest consensus state: current_term: 1 leader_uuid: "a91cf74c5953459eb403cc3461b3784a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a91cf74c5953459eb403cc3461b3784a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42395 } } }
I20260929 19:01:03.289134 3404 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 19:01:03.289213 3398 sys_catalog.cc:458] T 00000000000000000000000000000000 P a91cf74c5953459eb403cc3461b3784a [sys.catalog]: This master's current role is: LEADER
I20260929 19:01:03.289639 3404 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 29 19:01:03 dist-test-slave-lqkv krb5kdc[3310](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708463, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 19:01:03.292567 3404 catalog_manager.cc:1347] Generated new cluster ID: e8e0510d3cb2455982dd66404ba75635
I20260929 19:01:03.292623 3404 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 19:01:03.299046 3395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:03.284411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43250 (local address 127.0.98.62:42395)
0929 19:01:03.284568 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:03.284577 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:03.284608 (+ 31us) server_negotiation.cc:407] Connection header received
0929 19:01:03.285103 (+ 495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:03.285109 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:03.285153 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:03.285293 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:01:03.285929 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:03.286760 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:03.287927 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:03.288583 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:03.288688 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:03.292177 (+ 3489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:01:03.292180 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:01:03.292181 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:01:03.292190 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:01:03.297460 (+ 5270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:03.297839 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:03.297842 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:03.297843 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:03.297947 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:03.298145 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:03.298146 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:03.298147 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:03.298449 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:01:03.298600 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0929 19:01:03.298699 (+ 99us) server_negotiation.cc:299] Negotiation successful
0929 19:01:03.298830 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":71,"threads_started":1}
I20260929 19:01:03.299906 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 19:01:03.309273 3404 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 19:01:03.309738 3404 catalog_manager.cc:1504] Loading token signing keys...
I20260929 19:01:03.314567 3404 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a91cf74c5953459eb403cc3461b3784a: Generated new TSK 0
I20260929 19:01:03.314742 3404 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 19:01:03.317179 3395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:03.313522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43262 (local address 127.0.98.62:42395)
0929 19:01:03.313561 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:03.313563 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:03.313708 (+ 145us) server_negotiation.cc:407] Connection header received
0929 19:01:03.313740 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:03.313741 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:03.313752 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:03.313811 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:01:03.314212 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:03.314992 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:03.315707 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:03.316030 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:03.316100 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:03.316716 (+ 616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:01:03.316717 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:01:03.316718 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 19:01:03.316940 (+ 222us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:01:03.316956 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 19:01:03.316963 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:01:03.317031 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0929 19:01:03.317080 (+ 49us) server_negotiation.cc:299] Negotiation successful
0929 19:01:03.317105 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260929 19:01:03.319262 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUbnLHE/build/release/bin/kudu with pid 3323
2026-09-29T19:01:03Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1292 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708447806542-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 19:01:03 dist-test-slave-lqkv krb5kdc[3424](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 19:01:03 dist-test-slave-lqkv krb5kdc[3424](info): set up 2 sockets
Sep 29 19:01:03 dist-test-slave-lqkv krb5kdc[3424](info): commencing operation
krb5kdc: starting...
W20260929 19:01:04.360996 392 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708447806542-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708447806542-392-0/krb5kdc/test-user.keytab.
Sep 29 19:01:04 dist-test-slave-lqkv krb5kdc[3424](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708464, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T19:01:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T19:01:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708447806542-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708447806542-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 19:01:04.471218 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUbnLHE/build/release/bin/kudu
/tmp/dist-test-taskUbnLHE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708447806542-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708447806542-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708447806542-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708447806542-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43997
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708447806542-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43997
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708447806542-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708447806542-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708447806542-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:01:04.548552 3437 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:01:04.548766 3437 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:01:04.548799 3437 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:01:04.549944 3437 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 19:01:04.550011 3437 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:01:04.550035 3437 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:01:04.550057 3437 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 19:01:04.550086 3437 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 19:01:04.551522 3437 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708447806542-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708447806542-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43997
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708447806542-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43997
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708447806542-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3437
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708447806542-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 18:09:13 UTC on ccycloud.quasar-hpkalf.root.comops.site
build id 4680913
I20260929 19:01:04.551779 3437 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:01:04.551997 3437 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:01:04.556560 3442 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:01:04.556542 3445 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:01:04.556674 3437 server_base.cc:1034] running on GCE node
W20260929 19:01:04.556582 3443 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:01:04.557113 3437 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:01:04.557341 3437 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:01:04.558476 3437 hybrid_clock.cc:648] HybridClock initialized: now 1790708464558457 us; error 40 us; skew 500 ppm
Sep 29 19:01:04 dist-test-slave-lqkv krb5kdc[3424](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708464, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:01:04.560889 3437 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 19:01:04.561280 3437 webserver.cc:460] Webserver started at http://127.0.98.62:36775/ using document root <none> and password file <none>
I20260929 19:01:04.561493 3437 fs_manager.cc:359] Metadata directory not provided
I20260929 19:01:04.561559 3437 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:01:04.561710 3437 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:01:04.562680 3437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708447806542-392-0/minicluster-data/master-0/data/instance:
uuid: "8f9fd7c67d7c4d27aad57e2160f50705"
format_stamp: "Formatted at 2026-09-29 19:01:04 on dist-test-slave-lqkv"
server_key: "95bd1d847c49f6c9a37643bfd50af135"
server_key_iv: "e74c850df0a79b77f82bd0b63e94101a"
server_key_version: "encryptionkey@0"
I20260929 19:01:04.563031 3437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708447806542-392-0/minicluster-data/master-0/wal/instance:
uuid: "8f9fd7c67d7c4d27aad57e2160f50705"
format_stamp: "Formatted at 2026-09-29 19:01:04 on dist-test-slave-lqkv"
server_key: "95bd1d847c49f6c9a37643bfd50af135"
server_key_iv: "e74c850df0a79b77f82bd0b63e94101a"
server_key_version: "encryptionkey@0"
I20260929 19:01:04.564316 3437 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 19:01:04.565102 3452 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:04.565265 3437 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 19:01:04.565361 3437 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708447806542-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708447806542-392-0/minicluster-data/master-0/wal
uuid: "8f9fd7c67d7c4d27aad57e2160f50705"
format_stamp: "Formatted at 2026-09-29 19:01:04 on dist-test-slave-lqkv"
server_key: "95bd1d847c49f6c9a37643bfd50af135"
server_key_iv: "e74c850df0a79b77f82bd0b63e94101a"
server_key_version: "encryptionkey@0"
I20260929 19:01:04.565441 3437 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708447806542-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708447806542-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708447806542-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:01:04.593395 3437 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:01:04.595189 3437 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:01:04.595356 3437 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:01:04.599336 3437 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43997
I20260929 19:01:04.599390 3504 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43997 every 8 connection(s)
I20260929 19:01:04.599756 3437 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708447806542-392-0/minicluster-data/master-0/data/info.pb
I20260929 19:01:04.600629 3505 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:01:04.602785 3505 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f9fd7c67d7c4d27aad57e2160f50705: Bootstrap starting.
I20260929 19:01:04.603355 3505 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8f9fd7c67d7c4d27aad57e2160f50705: Neither blocks nor log segments found. Creating new log.
I20260929 19:01:04.603628 3505 log.cc:824] T 00000000000000000000000000000000 P 8f9fd7c67d7c4d27aad57e2160f50705: Log is configured to *not* fsync() on all Append() calls
I20260929 19:01:04.604259 3505 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f9fd7c67d7c4d27aad57e2160f50705: No bootstrap required, opened a new log
I20260929 19:01:04.605242 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUbnLHE/build/release/bin/kudu as pid 3437
I20260929 19:01:04.605347 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708447806542-392-0/minicluster-data/master-0/wal/instance
I20260929 19:01:04.605535 392 external_mini_cluster.cc:1428] Setting key bf9737ae5663dce3895c6995ff20db1f
I20260929 19:01:04.606482 3505 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8f9fd7c67d7c4d27aad57e2160f50705 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f9fd7c67d7c4d27aad57e2160f50705" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43997 } }
I20260929 19:01:04.606729 3505 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8f9fd7c67d7c4d27aad57e2160f50705 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:01:04.606807 3505 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8f9fd7c67d7c4d27aad57e2160f50705 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f9fd7c67d7c4d27aad57e2160f50705, State: Initialized, Role: FOLLOWER
I20260929 19:01:04.606956 3505 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8f9fd7c67d7c4d27aad57e2160f50705 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f9fd7c67d7c4d27aad57e2160f50705" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43997 } }
I20260929 19:01:04.607043 3505 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8f9fd7c67d7c4d27aad57e2160f50705 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 19:01:04.607079 3505 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8f9fd7c67d7c4d27aad57e2160f50705 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 19:01:04.607136 3505 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8f9fd7c67d7c4d27aad57e2160f50705 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:01:04.608028 3505 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8f9fd7c67d7c4d27aad57e2160f50705 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f9fd7c67d7c4d27aad57e2160f50705" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43997 } }
I20260929 19:01:04.608114 3505 leader_election.cc:302] T 00000000000000000000000000000000 P 8f9fd7c67d7c4d27aad57e2160f50705 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8f9fd7c67d7c4d27aad57e2160f50705; no voters:
I20260929 19:01:04.608285 3505 leader_election.cc:288] T 00000000000000000000000000000000 P 8f9fd7c67d7c4d27aad57e2160f50705 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 19:01:04.608321 3510 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8f9fd7c67d7c4d27aad57e2160f50705 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:01:04.608485 3510 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8f9fd7c67d7c4d27aad57e2160f50705 [term 1 LEADER]: Becoming Leader. State: Replica: 8f9fd7c67d7c4d27aad57e2160f50705, State: Running, Role: LEADER
I20260929 19:01:04.608523 3505 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8f9fd7c67d7c4d27aad57e2160f50705 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 19:01:04.608611 3510 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8f9fd7c67d7c4d27aad57e2160f50705 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f9fd7c67d7c4d27aad57e2160f50705" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43997 } }
I20260929 19:01:04.608947 3510 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f9fd7c67d7c4d27aad57e2160f50705 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8f9fd7c67d7c4d27aad57e2160f50705. Latest consensus state: current_term: 1 leader_uuid: "8f9fd7c67d7c4d27aad57e2160f50705" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f9fd7c67d7c4d27aad57e2160f50705" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43997 } } }
I20260929 19:01:04.609050 3510 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f9fd7c67d7c4d27aad57e2160f50705 [sys.catalog]: This master's current role is: LEADER
I20260929 19:01:04.609196 3511 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f9fd7c67d7c4d27aad57e2160f50705 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8f9fd7c67d7c4d27aad57e2160f50705" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f9fd7c67d7c4d27aad57e2160f50705" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43997 } } }
I20260929 19:01:04.609292 3511 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f9fd7c67d7c4d27aad57e2160f50705 [sys.catalog]: This master's current role is: LEADER
I20260929 19:01:04.610299 3516 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 19:01:04.610798 3516 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 29 19:01:04 dist-test-slave-lqkv krb5kdc[3424](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708464, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 19:01:04.612752 3516 catalog_manager.cc:1347] Generated new cluster ID: d0f47118df83401685d8c967b8f71556
I20260929 19:01:04.612816 3516 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 19:01:04.616326 3509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:04.606037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45972 (local address 127.0.98.62:43997)
0929 19:01:04.606230 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:04.606237 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:04.606260 (+ 23us) server_negotiation.cc:407] Connection header received
0929 19:01:04.606449 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:04.606454 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:04.606493 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:04.606593 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:01:04.607413 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:04.608394 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:04.609505 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:04.610033 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:04.610134 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:04.612744 (+ 2610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:01:04.612748 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:01:04.612749 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:01:04.612760 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:01:04.614717 (+ 1957us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:04.615106 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:04.615109 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:04.615109 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:04.615239 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:04.615441 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:04.615442 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:04.615442 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:04.615738 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:01:04.615901 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0929 19:01:04.615970 (+ 69us) server_negotiation.cc:299] Negotiation successful
0929 19:01:04.616099 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":97,"threads_started":1}
I20260929 19:01:04.622383 3516 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 19:01:04.622856 3516 catalog_manager.cc:1504] Loading token signing keys...
I20260929 19:01:04.628894 3516 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8f9fd7c67d7c4d27aad57e2160f50705: Generated new TSK 0
I20260929 19:01:04.629060 3516 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708447806542-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708447806542-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260929 19:01:04.653573 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUbnLHE/build/release/bin/kudu
/tmp/dist-test-taskUbnLHE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708447806542-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708447806542-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708447806542-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708447806542-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43997
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708447806542-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708447806542-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708447806542-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708447806542-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:01:04.732566 3531 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:01:04.732750 3531 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:01:04.732779 3531 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:01:04.733912 3531 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:01:04.733952 3531 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:01:04.733978 3531 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 19:01:04.735306 3531 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708447806542-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708447806542-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708447806542-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708447806542-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43997
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3531
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708447806542-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 18:09:13 UTC on ccycloud.quasar-hpkalf.root.comops.site
build id 4680913
I20260929 19:01:04.735526 3531 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:01:04.735713 3531 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:01:04.740504 3536 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:01:04.740566 3537 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:01:04.740689 3539 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:01:04.740674 3531 server_base.cc:1034] running on GCE node
I20260929 19:01:04.740927 3531 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:01:04.741107 3531 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:01:04.742285 3531 hybrid_clock.cc:648] HybridClock initialized: now 1790708464742271 us; error 25 us; skew 500 ppm
Sep 29 19:01:04 dist-test-slave-lqkv krb5kdc[3424](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708464, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:01:04.744720 3531 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 19:01:04.745131 3531 webserver.cc:460] Webserver started at http://127.0.98.1:40013/ using document root <none> and password file <none>
I20260929 19:01:04.745342 3531 fs_manager.cc:359] Metadata directory not provided
I20260929 19:01:04.745414 3531 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:01:04.745558 3531 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:01:04.746590 3531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708447806542-392-0/minicluster-data/ts-0/data/instance:
uuid: "46884a7f188b4751aea2eb296f9ff60c"
format_stamp: "Formatted at 2026-09-29 19:01:04 on dist-test-slave-lqkv"
server_key: "9f075c59a6317be58c91cb65c9b494c1"
server_key_iv: "120087cf3255db4190e301df6917ff9c"
server_key_version: "encryptionkey@0"
I20260929 19:01:04.746976 3531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708447806542-392-0/minicluster-data/ts-0/wal/instance:
uuid: "46884a7f188b4751aea2eb296f9ff60c"
format_stamp: "Formatted at 2026-09-29 19:01:04 on dist-test-slave-lqkv"
server_key: "9f075c59a6317be58c91cb65c9b494c1"
server_key_iv: "120087cf3255db4190e301df6917ff9c"
server_key_version: "encryptionkey@0"
I20260929 19:01:04.748276 3531 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 19:01:04.749061 3546 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:04.749317 3531 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 19:01:04.749411 3531 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708447806542-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708447806542-392-0/minicluster-data/ts-0/wal
uuid: "46884a7f188b4751aea2eb296f9ff60c"
format_stamp: "Formatted at 2026-09-29 19:01:04 on dist-test-slave-lqkv"
server_key: "9f075c59a6317be58c91cb65c9b494c1"
server_key_iv: "120087cf3255db4190e301df6917ff9c"
server_key_version: "encryptionkey@0"
I20260929 19:01:04.749486 3531 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708447806542-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708447806542-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708447806542-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:01:04.768021 3531 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:01:04.770278 3531 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:01:04.770430 3531 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:01:04.770635 3531 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:01:04.770947 3531 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:01:04.770998 3531 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:04.771034 3531 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:01:04.771061 3531 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:04.776768 3531 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42593
I20260929 19:01:04.776829 3659 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42593 every 8 connection(s)
I20260929 19:01:04.777100 3531 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708447806542-392-0/minicluster-data/ts-0/data/info.pb
I20260929 19:01:04.777869 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUbnLHE/build/release/bin/kudu as pid 3531
I20260929 19:01:04.777971 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708447806542-392-0/minicluster-data/ts-0/wal/instance
I20260929 19:01:04.778146 392 external_mini_cluster.cc:1428] Setting key b52d76738c1b51cfa6bbe14fe39ebeeb
Sep 29 19:01:04 dist-test-slave-lqkv krb5kdc[3424](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708464, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260929 19:01:04.786618 3509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:04.777703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34563 (local address 127.0.98.62:43997)
0929 19:01:04.777775 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:04.777777 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:04.777980 (+ 203us) server_negotiation.cc:407] Connection header received
0929 19:01:04.778523 (+ 543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:04.778524 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:04.778538 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:04.778597 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:01:04.779467 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:04.780181 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:04.781284 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:04.781632 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:04.781714 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:04.783769 (+ 2055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:01:04.783770 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:01:04.783770 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:01:04.783778 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:01:04.785397 (+ 1619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:04.785753 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:04.785754 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:04.785754 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:04.785811 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:04.786073 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:04.786073 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:04.786074 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:04.786212 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:01:04.786282 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 19:01:04.786493 (+ 211us) server_negotiation.cc:299] Negotiation successful
0929 19:01:04.786558 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 19:01:04.786774 3662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:04.777687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43997 (local address 127.0.98.1:34563)
0929 19:01:04.777845 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:04.777886 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 19:01:04.778425 (+ 539us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:01:04.778662 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:01:04.778666 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:01:04.779195 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:01:04.779371 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:04.779375 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:04.780284 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:04.780286 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:04.781202 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:04.781204 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:04.781714 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:04.781714 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:04.781826 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:04.782373 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:01:04.782381 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:01:04.783678 (+ 1297us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:01:04.785477 (+ 1799us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:01:04.785482 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:01:04.785483 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:01:04.785682 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:01:04.785885 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:01:04.785885 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:01:04.785886 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:01:04.786016 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:01:04.786277 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:01:04.786281 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:01:04.786384 (+ 103us) client_negotiation.cc:769] Sending connection context
0929 19:01:04.786461 (+ 77us) client_negotiation.cc:240] Negotiation successful
0929 19:01:04.786565 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":93,"threads_started":1}
I20260929 19:01:04.787277 3660 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43997
I20260929 19:01:04.787366 3660 heartbeater.cc:464] Registering TS with master...
I20260929 19:01:04.787539 3660 heartbeater.cc:511] Master 127.0.98.62:43997 requested a full tablet report, sending...
I20260929 19:01:04.788170 3469 ts_manager.cc:194] Registered new tserver with Master: 46884a7f188b4751aea2eb296f9ff60c (127.0.98.1:42593)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260929 19:01:04.789016 3469 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:34563
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708447806542-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708447806542-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260929 19:01:04.801486 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUbnLHE/build/release/bin/kudu
/tmp/dist-test-taskUbnLHE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708447806542-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708447806542-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708447806542-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708447806542-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43997
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708447806542-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708447806542-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708447806542-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708447806542-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:01:04.880285 3665 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:01:04.880461 3665 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:01:04.880482 3665 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:01:04.881524 3665 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:01:04.881559 3665 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:01:04.881583 3665 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 19:01:04.882840 3665 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708447806542-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708447806542-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708447806542-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708447806542-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43997
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3665
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708447806542-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 18:09:13 UTC on ccycloud.quasar-hpkalf.root.comops.site
build id 4680913
I20260929 19:01:04.882999 3665 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:01:04.883147 3665 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:01:04.887742 3671 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:01:04.887713 3673 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:01:04.887719 3670 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:01:04.887943 3665 server_base.cc:1034] running on GCE node
I20260929 19:01:04.888166 3665 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:01:04.888392 3665 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:01:04.889564 3665 hybrid_clock.cc:648] HybridClock initialized: now 1790708464889543 us; error 29 us; skew 500 ppm
Sep 29 19:01:04 dist-test-slave-lqkv krb5kdc[3424](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708464, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:01:04.891947 3665 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 19:01:04.892416 3665 webserver.cc:460] Webserver started at http://127.0.98.2:44517/ using document root <none> and password file <none>
I20260929 19:01:04.892633 3665 fs_manager.cc:359] Metadata directory not provided
I20260929 19:01:04.892699 3665 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:01:04.892877 3665 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:01:04.893870 3665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708447806542-392-0/minicluster-data/ts-1/data/instance:
uuid: "a44e8711f9104387bf7a9e2708994803"
format_stamp: "Formatted at 2026-09-29 19:01:04 on dist-test-slave-lqkv"
server_key: "29fc0e705176384cb390dd7fa1d946d5"
server_key_iv: "be87094788b35eccd48bb57ead9a1854"
server_key_version: "encryptionkey@0"
I20260929 19:01:04.894223 3665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708447806542-392-0/minicluster-data/ts-1/wal/instance:
uuid: "a44e8711f9104387bf7a9e2708994803"
format_stamp: "Formatted at 2026-09-29 19:01:04 on dist-test-slave-lqkv"
server_key: "29fc0e705176384cb390dd7fa1d946d5"
server_key_iv: "be87094788b35eccd48bb57ead9a1854"
server_key_version: "encryptionkey@0"
I20260929 19:01:04.895509 3665 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 19:01:04.896262 3680 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:04.896431 3665 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 19:01:04.896526 3665 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708447806542-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708447806542-392-0/minicluster-data/ts-1/wal
uuid: "a44e8711f9104387bf7a9e2708994803"
format_stamp: "Formatted at 2026-09-29 19:01:04 on dist-test-slave-lqkv"
server_key: "29fc0e705176384cb390dd7fa1d946d5"
server_key_iv: "be87094788b35eccd48bb57ead9a1854"
server_key_version: "encryptionkey@0"
I20260929 19:01:04.896605 3665 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708447806542-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708447806542-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708447806542-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:01:04.912336 3665 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:01:04.914070 3665 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:01:04.914227 3665 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:01:04.914436 3665 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:01:04.914726 3665 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:01:04.914788 3665 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:04.914826 3665 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:01:04.914855 3665 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:04.920521 3665 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36737
I20260929 19:01:04.920565 3793 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36737 every 8 connection(s)
I20260929 19:01:04.920864 3665 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708447806542-392-0/minicluster-data/ts-1/data/info.pb
I20260929 19:01:04.925673 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUbnLHE/build/release/bin/kudu as pid 3665
I20260929 19:01:04.925781 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708447806542-392-0/minicluster-data/ts-1/wal/instance
I20260929 19:01:04.925985 392 external_mini_cluster.cc:1428] Setting key 03d6245a7b5c126699baf7558bf36cff
Sep 29 19:01:04 dist-test-slave-lqkv krb5kdc[3424](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708464, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 19:01:04.930222 3796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:04.921466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43997 (local address 127.0.98.2:50415)
0929 19:01:04.921635 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:04.921667 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0929 19:01:04.922147 (+ 480us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:01:04.922429 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:01:04.922432 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:01:04.922961 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:01:04.923130 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:04.923134 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:04.923847 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:04.923848 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:04.924616 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:04.924617 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:04.925079 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:04.925079 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:04.925196 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:04.925631 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:01:04.925642 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:01:04.927213 (+ 1571us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:01:04.928907 (+ 1694us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:01:04.928912 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:01:04.928913 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:01:04.929165 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:01:04.929383 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:01:04.929383 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:01:04.929383 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:01:04.929469 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:01:04.929754 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:01:04.929757 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:01:04.929871 (+ 114us) client_negotiation.cc:769] Sending connection context
0929 19:01:04.929945 (+ 74us) client_negotiation.cc:240] Negotiation successful
0929 19:01:04.930068 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":95,"threads_started":1}
I20260929 19:01:04.930218 3509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:04.921475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50415 (local address 127.0.98.62:43997)
0929 19:01:04.921541 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:04.921543 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:04.921750 (+ 207us) server_negotiation.cc:407] Connection header received
0929 19:01:04.922292 (+ 542us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:04.922294 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:04.922306 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:04.922361 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:01:04.923230 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:04.923751 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:04.924705 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:04.924988 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:04.925071 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:04.927310 (+ 2239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:01:04.927311 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:01:04.927311 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:01:04.927319 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:01:04.928743 (+ 1424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:04.929242 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:04.929243 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:04.929243 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:04.929317 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:04.929531 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:04.929532 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:04.929532 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:04.929690 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:01:04.929761 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 19:01:04.929994 (+ 233us) server_negotiation.cc:299] Negotiation successful
0929 19:01:04.930099 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 19:01:04.930871 3794 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43997
I20260929 19:01:04.930976 3794 heartbeater.cc:464] Registering TS with master...
I20260929 19:01:04.931137 3794 heartbeater.cc:511] Master 127.0.98.62:43997 requested a full tablet report, sending...
I20260929 19:01:04.931494 3469 ts_manager.cc:194] Registered new tserver with Master: a44e8711f9104387bf7a9e2708994803 (127.0.98.2:36737)
I20260929 19:01:04.932018 3469 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:50415
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708447806542-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708447806542-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260929 19:01:04.951488 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUbnLHE/build/release/bin/kudu
/tmp/dist-test-taskUbnLHE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708447806542-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708447806542-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708447806542-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708447806542-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43997
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708447806542-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708447806542-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708447806542-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708447806542-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:01:05.030387 3799 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:01:05.030575 3799 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:01:05.030598 3799 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:01:05.031730 3799 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:01:05.031771 3799 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:01:05.031796 3799 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 19:01:05.033200 3799 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708447806542-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708447806542-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708447806542-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708447806542-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43997
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3799
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708447806542-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 18:09:13 UTC on ccycloud.quasar-hpkalf.root.comops.site
build id 4680913
I20260929 19:01:05.033442 3799 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:01:05.033689 3799 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:01:05.038364 3805 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:01:05.038380 3807 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:01:05.038476 3799 server_base.cc:1034] running on GCE node
W20260929 19:01:05.038424 3804 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:01:05.038735 3799 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:01:05.038973 3799 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:01:05.040110 3799 hybrid_clock.cc:648] HybridClock initialized: now 1790708465040095 us; error 38 us; skew 500 ppm
Sep 29 19:01:05 dist-test-slave-lqkv krb5kdc[3424](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708465, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:01:05.042675 3799 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 19:01:05.043133 3799 webserver.cc:460] Webserver started at http://127.0.98.3:38139/ using document root <none> and password file <none>
I20260929 19:01:05.043366 3799 fs_manager.cc:359] Metadata directory not provided
I20260929 19:01:05.043411 3799 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:01:05.043680 3799 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:01:05.044771 3799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708447806542-392-0/minicluster-data/ts-2/data/instance:
uuid: "f1f49f4765b64df7aa6655dce35660d2"
format_stamp: "Formatted at 2026-09-29 19:01:05 on dist-test-slave-lqkv"
server_key: "df4f61079b07e4adf89d410131c61732"
server_key_iv: "9b176f1630b88a4fec28847dd4882137"
server_key_version: "encryptionkey@0"
I20260929 19:01:05.045194 3799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708447806542-392-0/minicluster-data/ts-2/wal/instance:
uuid: "f1f49f4765b64df7aa6655dce35660d2"
format_stamp: "Formatted at 2026-09-29 19:01:05 on dist-test-slave-lqkv"
server_key: "df4f61079b07e4adf89d410131c61732"
server_key_iv: "9b176f1630b88a4fec28847dd4882137"
server_key_version: "encryptionkey@0"
I20260929 19:01:05.046464 3799 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 19:01:05.047163 3814 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:05.047358 3799 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 19:01:05.047431 3799 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708447806542-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708447806542-392-0/minicluster-data/ts-2/wal
uuid: "f1f49f4765b64df7aa6655dce35660d2"
format_stamp: "Formatted at 2026-09-29 19:01:05 on dist-test-slave-lqkv"
server_key: "df4f61079b07e4adf89d410131c61732"
server_key_iv: "9b176f1630b88a4fec28847dd4882137"
server_key_version: "encryptionkey@0"
I20260929 19:01:05.047479 3799 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708447806542-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708447806542-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708447806542-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:01:05.084143 3799 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:01:05.086041 3799 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:01:05.086146 3799 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:01:05.086335 3799 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:01:05.086634 3799 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:01:05.086666 3799 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:05.086688 3799 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:01:05.086747 3799 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:05.091950 3799 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41593
I20260929 19:01:05.092018 3927 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41593 every 8 connection(s)
I20260929 19:01:05.092329 3799 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708447806542-392-0/minicluster-data/ts-2/data/info.pb
I20260929 19:01:05.096163 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUbnLHE/build/release/bin/kudu as pid 3799
I20260929 19:01:05.096257 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790708447806542-392-0/minicluster-data/ts-2/wal/instance
I20260929 19:01:05.096503 392 external_mini_cluster.cc:1428] Setting key f5654b2db12dce87d2b76b2b1bec3d18
Sep 29 19:01:05 dist-test-slave-lqkv krb5kdc[3424](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708465, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 19:01:05.101532 3509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:05.093040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43369 (local address 127.0.98.62:43997)
0929 19:01:05.093101 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:05.093103 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:05.093283 (+ 180us) server_negotiation.cc:407] Connection header received
0929 19:01:05.093807 (+ 524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:05.093808 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:05.093820 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:05.093862 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:01:05.094775 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:05.095266 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:05.096473 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:05.096765 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:05.096866 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:05.098906 (+ 2040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:01:05.098908 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:01:05.098908 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:01:05.098914 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:01:05.100285 (+ 1371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:05.100683 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:05.100684 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:05.100684 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:05.100732 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:05.101004 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:05.101004 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:05.101005 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:05.101101 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:01:05.101151 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0929 19:01:05.101414 (+ 263us) server_negotiation.cc:299] Negotiation successful
0929 19:01:05.101471 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260929 19:01:05.101701 3930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:05.093026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43997 (local address 127.0.98.3:43369)
0929 19:01:05.093173 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:05.093205 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0929 19:01:05.093690 (+ 485us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:01:05.093938 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:01:05.093941 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:01:05.094482 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:01:05.094658 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:05.094669 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:05.095372 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:05.095373 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:05.096387 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:05.096389 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:05.096872 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:05.096873 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:05.097000 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:05.097448 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:01:05.097456 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:01:05.098832 (+ 1376us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:01:05.100354 (+ 1522us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:01:05.100360 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:01:05.100362 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:01:05.100619 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:01:05.100790 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:01:05.100791 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:01:05.100791 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:01:05.100949 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:01:05.101154 (+ 205us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:01:05.101176 (+ 22us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:01:05.101301 (+ 125us) client_negotiation.cc:769] Sending connection context
0929 19:01:05.101380 (+ 79us) client_negotiation.cc:240] Negotiation successful
0929 19:01:05.101488 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":74,"threads_started":1}
I20260929 19:01:05.102202 3928 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43997
I20260929 19:01:05.102272 3928 heartbeater.cc:464] Registering TS with master...
I20260929 19:01:05.102473 3928 heartbeater.cc:511] Master 127.0.98.62:43997 requested a full tablet report, sending...
I20260929 19:01:05.102833 3469 ts_manager.cc:194] Registered new tserver with Master: f1f49f4765b64df7aa6655dce35660d2 (127.0.98.3:41593)
I20260929 19:01:05.103171 3469 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:43369
I20260929 19:01:05.108372 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 19:01:05 dist-test-slave-lqkv krb5kdc[3424](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708465, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 29 19:01:05 dist-test-slave-lqkv krb5kdc[3424](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708465, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 19:01:05.130175 3509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:05.123053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45988 (local address 127.0.98.62:43997)
0929 19:01:05.123134 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:05.123136 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:05.123223 (+ 87us) server_negotiation.cc:407] Connection header received
0929 19:01:05.123327 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:05.123328 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:05.123339 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:05.123400 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:01:05.123915 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:05.124424 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:05.125252 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:05.125446 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:05.125528 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:05.127802 (+ 2274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:01:05.127804 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:01:05.127804 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:01:05.127811 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:01:05.129093 (+ 1282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:05.129422 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:05.129423 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:05.129423 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:05.129478 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:05.129667 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:05.129668 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:05.129668 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:05.129807 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:01:05.129911 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 19:01:05.130003 (+ 92us) server_negotiation.cc:299] Negotiation successful
0929 19:01:05.130075 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
W20260929 19:01:05.130409 3469 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:45988
I20260929 19:01:05.130822 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUbnLHE/build/release/bin/kudu with pid 3531
I20260929 19:01:05.134856 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUbnLHE/build/release/bin/kudu with pid 3665
I20260929 19:01:05.139073 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUbnLHE/build/release/bin/kudu with pid 3799
I20260929 19:01:05.143296 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUbnLHE/build/release/bin/kudu with pid 3437
2026-09-29T19:01:05Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1826 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708447806542-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 19:01:05 dist-test-slave-lqkv krb5kdc[3940](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 19:01:05 dist-test-slave-lqkv krb5kdc[3940](info): set up 2 sockets
Sep 29 19:01:05 dist-test-slave-lqkv krb5kdc[3940](info): commencing operation
krb5kdc: starting...
W20260929 19:01:06.173022 392 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708447806542-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708447806542-392-0/krb5kdc/test-user.keytab.
Sep 29 19:01:06 dist-test-slave-lqkv krb5kdc[3940](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708466, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T19:01:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T19:01:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708447806542-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708447806542-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 19:01:06.283490 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUbnLHE/build/release/bin/kudu
/tmp/dist-test-taskUbnLHE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708447806542-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708447806542-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708447806542-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708447806542-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36753
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708447806542-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36753
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708447806542-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708447806542-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708447806542-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:01:06.361347 3953 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:01:06.361637 3953 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:01:06.361685 3953 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:01:06.362926 3953 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 19:01:06.362985 3953 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:01:06.363008 3953 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:01:06.363037 3953 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 19:01:06.363061 3953 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 19:01:06.364545 3953 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708447806542-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708447806542-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36753
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708447806542-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36753
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708447806542-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3953
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708447806542-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 18:09:13 UTC on ccycloud.quasar-hpkalf.root.comops.site
build id 4680913
I20260929 19:01:06.364845 3953 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:01:06.365073 3953 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:01:06.369735 3961 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:01:06.369839 3959 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:01:06.369743 3958 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:01:06.370095 3953 server_base.cc:1034] running on GCE node
I20260929 19:01:06.370378 3953 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:01:06.370692 3953 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:01:06.371877 3953 hybrid_clock.cc:648] HybridClock initialized: now 1790708466371854 us; error 46 us; skew 500 ppm
Sep 29 19:01:06 dist-test-slave-lqkv krb5kdc[3940](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708466, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:01:06.386844 3953 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 19:01:06.387271 3953 webserver.cc:460] Webserver started at http://127.0.98.62:32903/ using document root <none> and password file <none>
I20260929 19:01:06.387470 3953 fs_manager.cc:359] Metadata directory not provided
I20260929 19:01:06.387509 3953 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:01:06.387593 3953 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:01:06.388592 3953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708447806542-392-0/minicluster-data/master-0/data/instance:
uuid: "a4d6718b7fb347d8b830cf1a2bd5af17"
format_stamp: "Formatted at 2026-09-29 19:01:06 on dist-test-slave-lqkv"
server_key: "0ce9d8b640e942eba5ff7c7aee2b8eea"
server_key_iv: "1625625863d8e16e568030f83e653985"
server_key_version: "encryptionkey@0"
I20260929 19:01:06.388952 3953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708447806542-392-0/minicluster-data/master-0/wal/instance:
uuid: "a4d6718b7fb347d8b830cf1a2bd5af17"
format_stamp: "Formatted at 2026-09-29 19:01:06 on dist-test-slave-lqkv"
server_key: "0ce9d8b640e942eba5ff7c7aee2b8eea"
server_key_iv: "1625625863d8e16e568030f83e653985"
server_key_version: "encryptionkey@0"
I20260929 19:01:06.390199 3953 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260929 19:01:06.390902 3968 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:06.391108 3953 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 19:01:06.391197 3953 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708447806542-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708447806542-392-0/minicluster-data/master-0/wal
uuid: "a4d6718b7fb347d8b830cf1a2bd5af17"
format_stamp: "Formatted at 2026-09-29 19:01:06 on dist-test-slave-lqkv"
server_key: "0ce9d8b640e942eba5ff7c7aee2b8eea"
server_key_iv: "1625625863d8e16e568030f83e653985"
server_key_version: "encryptionkey@0"
I20260929 19:01:06.391250 3953 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708447806542-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708447806542-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708447806542-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:01:06.405735 3953 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:01:06.407557 3953 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:01:06.407665 3953 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:01:06.411286 3953 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36753
I20260929 19:01:06.411341 4020 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36753 every 8 connection(s)
I20260929 19:01:06.411660 3953 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708447806542-392-0/minicluster-data/master-0/data/info.pb
I20260929 19:01:06.412484 4021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:01:06.414659 4021 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4d6718b7fb347d8b830cf1a2bd5af17: Bootstrap starting.
I20260929 19:01:06.415215 4021 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a4d6718b7fb347d8b830cf1a2bd5af17: Neither blocks nor log segments found. Creating new log.
I20260929 19:01:06.415493 4021 log.cc:824] T 00000000000000000000000000000000 P a4d6718b7fb347d8b830cf1a2bd5af17: Log is configured to *not* fsync() on all Append() calls
I20260929 19:01:06.416119 4021 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4d6718b7fb347d8b830cf1a2bd5af17: No bootstrap required, opened a new log
I20260929 19:01:06.417492 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUbnLHE/build/release/bin/kudu as pid 3953
I20260929 19:01:06.417580 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708447806542-392-0/minicluster-data/master-0/wal/instance
I20260929 19:01:06.417759 392 external_mini_cluster.cc:1428] Setting key 26c3f29c6ac368c18fd55650c401a4c0
I20260929 19:01:06.417845 4021 raft_consensus.cc:348] T 00000000000000000000000000000000 P a4d6718b7fb347d8b830cf1a2bd5af17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4d6718b7fb347d8b830cf1a2bd5af17" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36753 } }
I20260929 19:01:06.418011 4021 raft_consensus.cc:374] T 00000000000000000000000000000000 P a4d6718b7fb347d8b830cf1a2bd5af17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:01:06.418072 4021 raft_consensus.cc:729] T 00000000000000000000000000000000 P a4d6718b7fb347d8b830cf1a2bd5af17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4d6718b7fb347d8b830cf1a2bd5af17, State: Initialized, Role: FOLLOWER
I20260929 19:01:06.418202 4021 consensus_queue.cc:260] T 00000000000000000000000000000000 P a4d6718b7fb347d8b830cf1a2bd5af17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4d6718b7fb347d8b830cf1a2bd5af17" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36753 } }
I20260929 19:01:06.418282 4021 raft_consensus.cc:388] T 00000000000000000000000000000000 P a4d6718b7fb347d8b830cf1a2bd5af17 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 19:01:06.418319 4021 raft_consensus.cc:482] T 00000000000000000000000000000000 P a4d6718b7fb347d8b830cf1a2bd5af17 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 19:01:06.418357 4021 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a4d6718b7fb347d8b830cf1a2bd5af17 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:01:06.419449 4021 raft_consensus.cc:504] T 00000000000000000000000000000000 P a4d6718b7fb347d8b830cf1a2bd5af17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4d6718b7fb347d8b830cf1a2bd5af17" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36753 } }
I20260929 19:01:06.419564 4021 leader_election.cc:302] T 00000000000000000000000000000000 P a4d6718b7fb347d8b830cf1a2bd5af17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a4d6718b7fb347d8b830cf1a2bd5af17; no voters:
I20260929 19:01:06.419730 4021 leader_election.cc:288] T 00000000000000000000000000000000 P a4d6718b7fb347d8b830cf1a2bd5af17 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 19:01:06.419796 4026 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a4d6718b7fb347d8b830cf1a2bd5af17 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:01:06.419925 4026 raft_consensus.cc:686] T 00000000000000000000000000000000 P a4d6718b7fb347d8b830cf1a2bd5af17 [term 1 LEADER]: Becoming Leader. State: Replica: a4d6718b7fb347d8b830cf1a2bd5af17, State: Running, Role: LEADER
I20260929 19:01:06.419981 4021 sys_catalog.cc:564] T 00000000000000000000000000000000 P a4d6718b7fb347d8b830cf1a2bd5af17 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 19:01:06.420001 4026 consensus_queue.cc:237] T 00000000000000000000000000000000 P a4d6718b7fb347d8b830cf1a2bd5af17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4d6718b7fb347d8b830cf1a2bd5af17" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36753 } }
I20260929 19:01:06.420322 4026 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4d6718b7fb347d8b830cf1a2bd5af17 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a4d6718b7fb347d8b830cf1a2bd5af17. Latest consensus state: current_term: 1 leader_uuid: "a4d6718b7fb347d8b830cf1a2bd5af17" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4d6718b7fb347d8b830cf1a2bd5af17" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36753 } } }
I20260929 19:01:06.420404 4026 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4d6718b7fb347d8b830cf1a2bd5af17 [sys.catalog]: This master's current role is: LEADER
I20260929 19:01:06.420341 4028 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4d6718b7fb347d8b830cf1a2bd5af17 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a4d6718b7fb347d8b830cf1a2bd5af17" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4d6718b7fb347d8b830cf1a2bd5af17" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36753 } } }
I20260929 19:01:06.420625 4028 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4d6718b7fb347d8b830cf1a2bd5af17 [sys.catalog]: This master's current role is: LEADER
I20260929 19:01:06.420610 4032 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 19:01:06.421602 4032 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 19:01:06.423292 4032 catalog_manager.cc:1347] Generated new cluster ID: bea05ff9d0da4223a939686c203a0413
I20260929 19:01:06.423347 4032 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 29 19:01:06 dist-test-slave-lqkv krb5kdc[3940](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708466, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 19:01:06.430622 4025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:06.418306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40174 (local address 127.0.98.62:36753)
0929 19:01:06.418476 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:06.418485 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:06.418521 (+ 36us) server_negotiation.cc:407] Connection header received
0929 19:01:06.418637 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:06.418642 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:06.418689 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:06.418795 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:01:06.419670 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:06.421138 (+ 1468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:06.422193 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:06.422757 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:06.422878 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:06.427078 (+ 4200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:01:06.427082 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:01:06.427082 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:01:06.427093 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:01:06.429033 (+ 1940us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:06.429444 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:06.429447 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:06.429447 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:06.429539 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:06.429748 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:06.429749 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:06.429749 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:06.430021 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:01:06.430182 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0929 19:01:06.430310 (+ 128us) server_negotiation.cc:299] Negotiation successful
0929 19:01:06.430432 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":75,"threads_started":1}
I20260929 19:01:06.439042 4032 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 19:01:06.439749 4032 catalog_manager.cc:1504] Loading token signing keys...
I20260929 19:01:06.452067 4032 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a4d6718b7fb347d8b830cf1a2bd5af17: Generated new TSK 0
I20260929 19:01:06.452250 4032 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708447806542-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708447806542-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260929 19:01:06.481897 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUbnLHE/build/release/bin/kudu
/tmp/dist-test-taskUbnLHE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708447806542-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708447806542-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708447806542-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708447806542-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36753
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708447806542-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708447806542-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708447806542-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708447806542-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:01:06.560832 4047 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:01:06.561044 4047 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:01:06.561080 4047 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:01:06.562263 4047 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:01:06.562327 4047 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:01:06.562373 4047 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 19:01:06.563840 4047 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708447806542-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708447806542-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708447806542-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708447806542-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36753
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4047
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708447806542-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 18:09:13 UTC on ccycloud.quasar-hpkalf.root.comops.site
build id 4680913
I20260929 19:01:06.564100 4047 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:01:06.564318 4047 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:01:06.568993 4052 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:01:06.569000 4055 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:01:06.569013 4053 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:01:06.569072 4047 server_base.cc:1034] running on GCE node
I20260929 19:01:06.569396 4047 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:01:06.569612 4047 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:01:06.570753 4047 hybrid_clock.cc:648] HybridClock initialized: now 1790708466570740 us; error 27 us; skew 500 ppm
Sep 29 19:01:06 dist-test-slave-lqkv krb5kdc[3940](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708466, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:01:06.573176 4047 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 19:01:06.573560 4047 webserver.cc:460] Webserver started at http://127.0.98.1:33391/ using document root <none> and password file <none>
I20260929 19:01:06.573766 4047 fs_manager.cc:359] Metadata directory not provided
I20260929 19:01:06.573832 4047 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:01:06.573974 4047 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:01:06.574928 4047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708447806542-392-0/minicluster-data/ts-0/data/instance:
uuid: "1430eb6b61204ed181a05dde1a4d5605"
format_stamp: "Formatted at 2026-09-29 19:01:06 on dist-test-slave-lqkv"
server_key: "411f63c45fd8908c91cfb6a4d944a7f7"
server_key_iv: "1021dc3407882f75375be1f0cedf3c7b"
server_key_version: "encryptionkey@0"
I20260929 19:01:06.575282 4047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708447806542-392-0/minicluster-data/ts-0/wal/instance:
uuid: "1430eb6b61204ed181a05dde1a4d5605"
format_stamp: "Formatted at 2026-09-29 19:01:06 on dist-test-slave-lqkv"
server_key: "411f63c45fd8908c91cfb6a4d944a7f7"
server_key_iv: "1021dc3407882f75375be1f0cedf3c7b"
server_key_version: "encryptionkey@0"
I20260929 19:01:06.576496 4047 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260929 19:01:06.577296 4062 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:06.577478 4047 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260929 19:01:06.577572 4047 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708447806542-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708447806542-392-0/minicluster-data/ts-0/wal
uuid: "1430eb6b61204ed181a05dde1a4d5605"
format_stamp: "Formatted at 2026-09-29 19:01:06 on dist-test-slave-lqkv"
server_key: "411f63c45fd8908c91cfb6a4d944a7f7"
server_key_iv: "1021dc3407882f75375be1f0cedf3c7b"
server_key_version: "encryptionkey@0"
I20260929 19:01:06.577646 4047 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708447806542-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708447806542-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708447806542-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:01:06.587075 4047 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:01:06.589231 4047 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:01:06.589385 4047 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:01:06.589593 4047 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:01:06.589870 4047 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:01:06.589922 4047 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:06.589962 4047 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:01:06.589986 4047 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:06.596167 4047 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37343
I20260929 19:01:06.596201 4175 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37343 every 8 connection(s)
I20260929 19:01:06.596488 4047 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708447806542-392-0/minicluster-data/ts-0/data/info.pb
Sep 29 19:01:06 dist-test-slave-lqkv krb5kdc[3940](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708466, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 19:01:06.605074 4025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:06.597213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49095 (local address 127.0.98.62:36753)
0929 19:01:06.597285 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:06.597287 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:06.597442 (+ 155us) server_negotiation.cc:407] Connection header received
0929 19:01:06.598297 (+ 855us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:06.598298 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:06.598310 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:06.598361 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:01:06.598900 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:06.599378 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:06.600290 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:06.600481 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:06.600530 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:06.602418 (+ 1888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:01:06.602419 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:01:06.602419 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:01:06.602426 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:01:06.603905 (+ 1479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:06.604236 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:06.604236 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:06.604237 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:06.604284 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:06.604502 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:06.604503 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:06.604503 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:06.604611 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:01:06.604675 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0929 19:01:06.604945 (+ 270us) server_negotiation.cc:299] Negotiation successful
0929 19:01:06.605012 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 19:01:06.605187 4178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:06.597173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36753 (local address 127.0.98.1:49095)
0929 19:01:06.597330 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:06.597362 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0929 19:01:06.598214 (+ 852us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:01:06.598429 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:01:06.598432 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:01:06.598657 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:01:06.598832 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:06.598836 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:06.599449 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:06.599450 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:06.600220 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:06.600222 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:06.600546 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:06.600546 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:06.600635 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:06.601060 (+ 425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:01:06.601068 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:01:06.602317 (+ 1249us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:01:06.603973 (+ 1656us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:01:06.603978 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:01:06.603979 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:01:06.604170 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:01:06.604339 (+ 169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:01:06.604340 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:01:06.604340 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:01:06.604426 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:01:06.604686 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:01:06.604689 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:01:06.604787 (+ 98us) client_negotiation.cc:769] Sending connection context
0929 19:01:06.604913 (+ 126us) client_negotiation.cc:240] Negotiation successful
0929 19:01:06.605021 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":88,"threads_started":1}
I20260929 19:01:06.605589 4176 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36753
I20260929 19:01:06.605660 4176 heartbeater.cc:464] Registering TS with master...
I20260929 19:01:06.605856 4176 heartbeater.cc:511] Master 127.0.98.62:36753 requested a full tablet report, sending...
I20260929 19:01:06.606331 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUbnLHE/build/release/bin/kudu as pid 4047
I20260929 19:01:06.606398 3985 ts_manager.cc:194] Registered new tserver with Master: 1430eb6b61204ed181a05dde1a4d5605 (127.0.98.1:37343)
I20260929 19:01:06.606437 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708447806542-392-0/minicluster-data/ts-0/wal/instance
I20260929 19:01:06.606674 392 external_mini_cluster.cc:1428] Setting key 6b3549ee75f2baa6bbe59c8ef36e8ddd
I20260929 19:01:06.607138 3985 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:49095
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708447806542-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708447806542-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260929 19:01:06.629112 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUbnLHE/build/release/bin/kudu
/tmp/dist-test-taskUbnLHE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708447806542-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708447806542-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708447806542-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708447806542-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36753
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708447806542-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708447806542-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708447806542-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708447806542-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:01:06.706619 4181 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:01:06.707031 4181 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:01:06.707072 4181 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:01:06.708300 4181 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:01:06.708374 4181 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:01:06.708419 4181 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 19:01:06.709939 4181 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708447806542-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708447806542-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708447806542-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708447806542-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36753
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4181
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708447806542-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 18:09:13 UTC on ccycloud.quasar-hpkalf.root.comops.site
build id 4680913
I20260929 19:01:06.710182 4181 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:01:06.710397 4181 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:01:06.714941 4187 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:01:06.714941 4189 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:01:06.715167 4181 server_base.cc:1034] running on GCE node
W20260929 19:01:06.715114 4186 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:01:06.715444 4181 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:01:06.715669 4181 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:01:06.716861 4181 hybrid_clock.cc:648] HybridClock initialized: now 1790708466716840 us; error 34 us; skew 500 ppm
Sep 29 19:01:06 dist-test-slave-lqkv krb5kdc[3940](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708466, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:01:06.719146 4181 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 19:01:06.719543 4181 webserver.cc:460] Webserver started at http://127.0.98.2:38403/ using document root <none> and password file <none>
I20260929 19:01:06.719810 4181 fs_manager.cc:359] Metadata directory not provided
I20260929 19:01:06.719919 4181 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:01:06.720016 4181 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:01:06.721115 4181 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708447806542-392-0/minicluster-data/ts-1/data/instance:
uuid: "1637354bd4b947ea82bf5c4046a46214"
format_stamp: "Formatted at 2026-09-29 19:01:06 on dist-test-slave-lqkv"
server_key: "7cfe2e7701037eeb5a981e8f822c4485"
server_key_iv: "71c6c15081f2c724203144e4c52fd373"
server_key_version: "encryptionkey@0"
I20260929 19:01:06.721393 4181 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708447806542-392-0/minicluster-data/ts-1/wal/instance:
uuid: "1637354bd4b947ea82bf5c4046a46214"
format_stamp: "Formatted at 2026-09-29 19:01:06 on dist-test-slave-lqkv"
server_key: "7cfe2e7701037eeb5a981e8f822c4485"
server_key_iv: "71c6c15081f2c724203144e4c52fd373"
server_key_version: "encryptionkey@0"
I20260929 19:01:06.722506 4181 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 19:01:06.723132 4196 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:06.723332 4181 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 19:01:06.723423 4181 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708447806542-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708447806542-392-0/minicluster-data/ts-1/wal
uuid: "1637354bd4b947ea82bf5c4046a46214"
format_stamp: "Formatted at 2026-09-29 19:01:06 on dist-test-slave-lqkv"
server_key: "7cfe2e7701037eeb5a981e8f822c4485"
server_key_iv: "71c6c15081f2c724203144e4c52fd373"
server_key_version: "encryptionkey@0"
I20260929 19:01:06.723503 4181 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708447806542-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708447806542-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708447806542-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:01:06.737481 4181 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:01:06.739042 4181 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:01:06.739187 4181 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:01:06.739377 4181 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:01:06.739640 4181 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:01:06.739703 4181 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:06.739741 4181 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:01:06.739769 4181 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:06.745527 4181 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39763
I20260929 19:01:06.745602 4309 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39763 every 8 connection(s)
I20260929 19:01:06.745855 4181 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708447806542-392-0/minicluster-data/ts-1/data/info.pb
Sep 29 19:01:06 dist-test-slave-lqkv krb5kdc[3940](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708466, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 19:01:06.752954 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUbnLHE/build/release/bin/kudu as pid 4181
I20260929 19:01:06.753044 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708447806542-392-0/minicluster-data/ts-1/wal/instance
I20260929 19:01:06.753225 392 external_mini_cluster.cc:1428] Setting key 56d4045d2b2954c170b234a5a8066eaf
I20260929 19:01:06.754532 4025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:06.746480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36837 (local address 127.0.98.62:36753)
0929 19:01:06.746537 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:06.746539 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:06.746736 (+ 197us) server_negotiation.cc:407] Connection header received
0929 19:01:06.747563 (+ 827us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:06.747564 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:06.747572 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:06.747624 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:01:06.748213 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:06.748688 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:06.749589 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:06.749776 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:06.749829 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:06.751667 (+ 1838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:01:06.751669 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:01:06.751669 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:01:06.751675 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:01:06.753157 (+ 1482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:06.753611 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:06.753612 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:06.753612 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:06.753675 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:06.753927 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:06.753929 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:06.753929 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:06.754044 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:01:06.754118 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 19:01:06.754367 (+ 249us) server_negotiation.cc:299] Negotiation successful
0929 19:01:06.754444 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260929 19:01:06.754731 4312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:06.746447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36753 (local address 127.0.98.2:36837)
0929 19:01:06.746608 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:06.746645 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 19:01:06.747480 (+ 835us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:01:06.747686 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:01:06.747689 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:01:06.747973 (+ 284us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:01:06.748141 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:06.748145 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:06.748757 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:06.748758 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:06.749519 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:06.749521 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:06.749846 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:06.749847 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:06.749941 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:06.750317 (+ 376us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:01:06.750325 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:01:06.751586 (+ 1261us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:01:06.753242 (+ 1656us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:01:06.753248 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:01:06.753250 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:01:06.753527 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:01:06.753741 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:01:06.753742 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:01:06.753742 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:01:06.753871 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:01:06.754110 (+ 239us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:01:06.754115 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:01:06.754244 (+ 129us) client_negotiation.cc:769] Sending connection context
0929 19:01:06.754346 (+ 102us) client_negotiation.cc:240] Negotiation successful
0929 19:01:06.754484 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":82,"threads_started":1}
I20260929 19:01:06.755115 4310 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36753
I20260929 19:01:06.755228 4310 heartbeater.cc:464] Registering TS with master...
I20260929 19:01:06.755389 4310 heartbeater.cc:511] Master 127.0.98.62:36753 requested a full tablet report, sending...
I20260929 19:01:06.755771 3985 ts_manager.cc:194] Registered new tserver with Master: 1637354bd4b947ea82bf5c4046a46214 (127.0.98.2:39763)
I20260929 19:01:06.756273 3985 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:36837
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708447806542-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708447806542-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260929 19:01:06.775516 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUbnLHE/build/release/bin/kudu
/tmp/dist-test-taskUbnLHE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708447806542-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708447806542-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708447806542-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708447806542-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36753
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708447806542-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36429
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708447806542-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708447806542-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708447806542-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:01:06.855150 4315 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:01:06.855356 4315 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:01:06.855392 4315 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:01:06.856496 4315 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:01:06.856556 4315 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:01:06.856600 4315 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 19:01:06.858043 4315 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36429
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708447806542-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708447806542-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708447806542-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708447806542-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36753
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4315
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708447806542-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 18:09:13 UTC on ccycloud.quasar-hpkalf.root.comops.site
build id 4680913
I20260929 19:01:06.858285 4315 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:01:06.858527 4315 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:01:06.863106 4323 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:01:06.863107 4320 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:01:06.863128 4321 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:01:06.863260 4315 server_base.cc:1034] running on GCE node
I20260929 19:01:06.863468 4315 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:01:06.863704 4315 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:01:06.864859 4315 hybrid_clock.cc:648] HybridClock initialized: now 1790708466864852 us; error 32 us; skew 500 ppm
Sep 29 19:01:06 dist-test-slave-lqkv krb5kdc[3940](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708466, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:01:06.867153 4315 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 19:01:06.867558 4315 webserver.cc:460] Webserver started at http://127.0.98.3:39003/ using document root <none> and password file <none>
I20260929 19:01:06.867728 4315 fs_manager.cc:359] Metadata directory not provided
I20260929 19:01:06.867765 4315 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:01:06.867856 4315 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:01:06.868703 4315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708447806542-392-0/minicluster-data/ts-2/data/instance:
uuid: "1d08011626da4d7b9282298bcab42334"
format_stamp: "Formatted at 2026-09-29 19:01:06 on dist-test-slave-lqkv"
server_key: "1b73463b76cb20d7850437566fa4057d"
server_key_iv: "b279296778f9ca72c7e5de39dc9ee507"
server_key_version: "encryptionkey@0"
I20260929 19:01:06.869046 4315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708447806542-392-0/minicluster-data/ts-2/wal/instance:
uuid: "1d08011626da4d7b9282298bcab42334"
format_stamp: "Formatted at 2026-09-29 19:01:06 on dist-test-slave-lqkv"
server_key: "1b73463b76cb20d7850437566fa4057d"
server_key_iv: "b279296778f9ca72c7e5de39dc9ee507"
server_key_version: "encryptionkey@0"
I20260929 19:01:06.870185 4315 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 19:01:06.870784 4330 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:06.870930 4315 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 19:01:06.871006 4315 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708447806542-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708447806542-392-0/minicluster-data/ts-2/wal
uuid: "1d08011626da4d7b9282298bcab42334"
format_stamp: "Formatted at 2026-09-29 19:01:06 on dist-test-slave-lqkv"
server_key: "1b73463b76cb20d7850437566fa4057d"
server_key_iv: "b279296778f9ca72c7e5de39dc9ee507"
server_key_version: "encryptionkey@0"
I20260929 19:01:06.871055 4315 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708447806542-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708447806542-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708447806542-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:01:06.896611 4315 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:01:06.898721 4315 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:01:06.898823 4315 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:01:06.898983 4315 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:01:06.899225 4315 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:01:06.899255 4315 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:06.899312 4315 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:01:06.899348 4315 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:06.904754 4315 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38789
I20260929 19:01:06.904847 4443 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38789 every 8 connection(s)
I20260929 19:01:06.905145 4315 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708447806542-392-0/minicluster-data/ts-2/data/info.pb
I20260929 19:01:06.909516 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUbnLHE/build/release/bin/kudu as pid 4315
I20260929 19:01:06.909598 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790708447806542-392-0/minicluster-data/ts-2/wal/instance
I20260929 19:01:06.909823 392 external_mini_cluster.cc:1428] Setting key 31596c115ce10afdaf2e1d7c458e2f57
Sep 29 19:01:06 dist-test-slave-lqkv krb5kdc[3940](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708466, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 19:01:06.913995 4025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:06.905816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42859 (local address 127.0.98.62:36753)
0929 19:01:06.905871 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:06.905873 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:06.906065 (+ 192us) server_negotiation.cc:407] Connection header received
0929 19:01:06.906984 (+ 919us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:06.906985 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:06.906996 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:06.907056 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:01:06.907624 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:06.908109 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:06.909190 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:06.909382 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:06.909466 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:06.911575 (+ 2109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:01:06.911576 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:01:06.911577 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:01:06.911584 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:01:06.912694 (+ 1110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:06.913093 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:06.913094 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:06.913094 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:06.913153 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:06.913391 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:06.913391 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:06.913392 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:06.913497 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:01:06.913573 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 19:01:06.913833 (+ 260us) server_negotiation.cc:299] Negotiation successful
0929 19:01:06.913909 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260929 19:01:06.914157 4446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:06.905795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36753 (local address 127.0.98.3:42859)
0929 19:01:06.905952 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:06.905990 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 19:01:06.906881 (+ 891us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:01:06.907122 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:01:06.907126 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:01:06.907367 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:01:06.907531 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:06.907534 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:06.908203 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:06.908204 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:06.909076 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:06.909078 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:06.909471 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:06.909472 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:06.909619 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:06.910091 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:01:06.910099 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:01:06.911485 (+ 1386us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:01:06.912775 (+ 1290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:01:06.912781 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:01:06.912783 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:01:06.913021 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:01:06.913229 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:01:06.913229 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:01:06.913230 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:01:06.913327 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:01:06.913564 (+ 237us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:01:06.913570 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:01:06.913700 (+ 130us) client_negotiation.cc:769] Sending connection context
0929 19:01:06.913795 (+ 95us) client_negotiation.cc:240] Negotiation successful
0929 19:01:06.913936 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":75,"threads_started":1}
I20260929 19:01:06.914613 4444 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36753
I20260929 19:01:06.914681 4444 heartbeater.cc:464] Registering TS with master...
I20260929 19:01:06.914848 4444 heartbeater.cc:511] Master 127.0.98.62:36753 requested a full tablet report, sending...
I20260929 19:01:06.915210 3985 ts_manager.cc:194] Registered new tserver with Master: 1d08011626da4d7b9282298bcab42334 (127.0.98.3:38789)
I20260929 19:01:06.915553 3985 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:42859
I20260929 19:01:06.921509 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 19:01:06 dist-test-slave-lqkv krb5kdc[3940](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708466, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 19:01:06 dist-test-slave-lqkv krb5kdc[3940](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708466, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260929 19:01:06.942528 4451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:06.935471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34642 (local address 127.0.98.1:37343)
0929 19:01:06.935607 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:06.935616 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:06.935625 (+ 9us) server_negotiation.cc:407] Connection header received
0929 19:01:06.935648 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:06.935649 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:06.935661 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:06.935722 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:01:06.936235 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:06.937005 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:06.937706 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:06.937988 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:06.938040 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:06.940017 (+ 1977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:01:06.940024 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:01:06.940024 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:01:06.940035 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:01:06.941353 (+ 1318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:06.941700 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:06.941703 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:06.941703 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:06.941757 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:06.941974 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:06.941975 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:06.941975 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:06.942095 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:01:06.942274 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0929 19:01:06.942340 (+ 66us) server_negotiation.cc:299] Negotiation successful
0929 19:01:06.942412 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":76,"threads_started":1}
Sep 29 19:01:06 dist-test-slave-lqkv krb5kdc[3940](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708466, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 19:01:06.949489 4025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:06.943611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40180 (local address 127.0.98.62:36753)
0929 19:01:06.943681 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:06.943683 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:06.943691 (+ 8us) server_negotiation.cc:407] Connection header received
0929 19:01:06.943728 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:06.943729 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:06.943737 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:06.943781 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:01:06.944177 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:06.944610 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:06.945267 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:06.945455 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:06.945517 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:06.947433 (+ 1916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:01:06.947434 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:01:06.947435 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:01:06.947440 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:01:06.948554 (+ 1114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:06.948887 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:06.948888 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:06.948888 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:06.948933 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:06.949076 (+ 143us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:06.949076 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:06.949077 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:06.949192 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:01:06.949256 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0929 19:01:06.949339 (+ 83us) server_negotiation.cc:299] Negotiation successful
0929 19:01:06.949390 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
W20260929 19:01:06.949705 3985 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40180
I20260929 19:01:06.950014 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUbnLHE/build/release/bin/kudu with pid 4047
I20260929 19:01:06.954661 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUbnLHE/build/release/bin/kudu with pid 4181
I20260929 19:01:06.958743 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUbnLHE/build/release/bin/kudu with pid 4315
I20260929 19:01:06.963235 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUbnLHE/build/release/bin/kudu with pid 3953
2026-09-29T19:01:06Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (1820 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708447806542-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 19:01:07 dist-test-slave-lqkv krb5kdc[4456](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 19:01:07 dist-test-slave-lqkv krb5kdc[4456](info): set up 2 sockets
Sep 29 19:01:07 dist-test-slave-lqkv krb5kdc[4456](info): commencing operation
krb5kdc: starting...
W20260929 19:01:08.000701 392 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708447806542-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708447806542-392-0/krb5kdc/test-user.keytab.
Sep 29 19:01:08 dist-test-slave-lqkv krb5kdc[4456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708468, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T19:01:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T19:01:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708447806542-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708447806542-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 19:01:08.109395 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUbnLHE/build/release/bin/kudu
/tmp/dist-test-taskUbnLHE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708447806542-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708447806542-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708447806542-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708447806542-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46623
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708447806542-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46623
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708447806542-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708447806542-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708447806542-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:01:08.186615 4469 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:01:08.186805 4469 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:01:08.186826 4469 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:01:08.187947 4469 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 19:01:08.187984 4469 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:01:08.187999 4469 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:01:08.188045 4469 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 19:01:08.188069 4469 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 19:01:08.189518 4469 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708447806542-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708447806542-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46623
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708447806542-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46623
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708447806542-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4469
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708447806542-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 18:09:13 UTC on ccycloud.quasar-hpkalf.root.comops.site
build id 4680913
I20260929 19:01:08.189755 4469 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:01:08.189963 4469 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:01:08.194362 4475 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:01:08.194450 4474 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:01:08.194505 4469 server_base.cc:1034] running on GCE node
W20260929 19:01:08.194583 4477 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:01:08.194795 4469 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:01:08.195065 4469 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:01:08.196199 4469 hybrid_clock.cc:648] HybridClock initialized: now 1790708468196189 us; error 27 us; skew 500 ppm
Sep 29 19:01:08 dist-test-slave-lqkv krb5kdc[4456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708468, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:01:08.198462 4469 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 19:01:08.198791 4469 fs_manager.cc:359] Metadata directory not provided
I20260929 19:01:08.198838 4469 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:01:08.198921 4469 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:01:08.199875 4469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708447806542-392-0/minicluster-data/master-0/data/instance:
uuid: "ffcc85f05b4c427e9cd78e6fc2dbb71b"
format_stamp: "Formatted at 2026-09-29 19:01:08 on dist-test-slave-lqkv"
server_key: "be708bf8ff39b0290e89ef1178c943c7"
server_key_iv: "51e2947ebd5a2830e4ba55bc57567a53"
server_key_version: "encryptionkey@0"
I20260929 19:01:08.200157 4469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708447806542-392-0/minicluster-data/master-0/wal/instance:
uuid: "ffcc85f05b4c427e9cd78e6fc2dbb71b"
format_stamp: "Formatted at 2026-09-29 19:01:08 on dist-test-slave-lqkv"
server_key: "be708bf8ff39b0290e89ef1178c943c7"
server_key_iv: "51e2947ebd5a2830e4ba55bc57567a53"
server_key_version: "encryptionkey@0"
I20260929 19:01:08.201385 4469 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 19:01:08.202061 4483 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:08.202236 4469 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 19:01:08.202301 4469 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708447806542-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708447806542-392-0/minicluster-data/master-0/wal
uuid: "ffcc85f05b4c427e9cd78e6fc2dbb71b"
format_stamp: "Formatted at 2026-09-29 19:01:08 on dist-test-slave-lqkv"
server_key: "be708bf8ff39b0290e89ef1178c943c7"
server_key_iv: "51e2947ebd5a2830e4ba55bc57567a53"
server_key_version: "encryptionkey@0"
I20260929 19:01:08.202349 4469 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708447806542-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708447806542-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708447806542-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:01:08.215333 4469 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:01:08.217087 4469 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:01:08.217188 4469 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:01:08.220659 4469 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46623
I20260929 19:01:08.220738 4535 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46623 every 8 connection(s)
I20260929 19:01:08.220996 4469 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708447806542-392-0/minicluster-data/master-0/data/info.pb
I20260929 19:01:08.221894 4536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:01:08.223191 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUbnLHE/build/release/bin/kudu as pid 4469
I20260929 19:01:08.223301 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708447806542-392-0/minicluster-data/master-0/wal/instance
I20260929 19:01:08.223491 392 external_mini_cluster.cc:1428] Setting key 945aa1d2d5139a0324a3c53b52e369ed
I20260929 19:01:08.224485 4536 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ffcc85f05b4c427e9cd78e6fc2dbb71b: Bootstrap starting.
I20260929 19:01:08.225226 4536 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ffcc85f05b4c427e9cd78e6fc2dbb71b: Neither blocks nor log segments found. Creating new log.
I20260929 19:01:08.225571 4536 log.cc:824] T 00000000000000000000000000000000 P ffcc85f05b4c427e9cd78e6fc2dbb71b: Log is configured to *not* fsync() on all Append() calls
I20260929 19:01:08.226208 4536 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ffcc85f05b4c427e9cd78e6fc2dbb71b: No bootstrap required, opened a new log
I20260929 19:01:08.228413 4536 raft_consensus.cc:348] T 00000000000000000000000000000000 P ffcc85f05b4c427e9cd78e6fc2dbb71b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffcc85f05b4c427e9cd78e6fc2dbb71b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46623 } }
I20260929 19:01:08.228591 4536 raft_consensus.cc:374] T 00000000000000000000000000000000 P ffcc85f05b4c427e9cd78e6fc2dbb71b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:01:08.228631 4536 raft_consensus.cc:729] T 00000000000000000000000000000000 P ffcc85f05b4c427e9cd78e6fc2dbb71b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ffcc85f05b4c427e9cd78e6fc2dbb71b, State: Initialized, Role: FOLLOWER
I20260929 19:01:08.228750 4536 consensus_queue.cc:260] T 00000000000000000000000000000000 P ffcc85f05b4c427e9cd78e6fc2dbb71b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffcc85f05b4c427e9cd78e6fc2dbb71b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46623 } }
I20260929 19:01:08.228861 4536 raft_consensus.cc:388] T 00000000000000000000000000000000 P ffcc85f05b4c427e9cd78e6fc2dbb71b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 19:01:08.228904 4536 raft_consensus.cc:482] T 00000000000000000000000000000000 P ffcc85f05b4c427e9cd78e6fc2dbb71b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 19:01:08.228963 4536 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ffcc85f05b4c427e9cd78e6fc2dbb71b [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:01:08.229660 4536 raft_consensus.cc:504] T 00000000000000000000000000000000 P ffcc85f05b4c427e9cd78e6fc2dbb71b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffcc85f05b4c427e9cd78e6fc2dbb71b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46623 } }
I20260929 19:01:08.229772 4536 leader_election.cc:302] T 00000000000000000000000000000000 P ffcc85f05b4c427e9cd78e6fc2dbb71b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ffcc85f05b4c427e9cd78e6fc2dbb71b; no voters:
I20260929 19:01:08.229928 4536 leader_election.cc:288] T 00000000000000000000000000000000 P ffcc85f05b4c427e9cd78e6fc2dbb71b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 19:01:08.229964 4541 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ffcc85f05b4c427e9cd78e6fc2dbb71b [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:01:08.230079 4541 raft_consensus.cc:686] T 00000000000000000000000000000000 P ffcc85f05b4c427e9cd78e6fc2dbb71b [term 1 LEADER]: Becoming Leader. State: Replica: ffcc85f05b4c427e9cd78e6fc2dbb71b, State: Running, Role: LEADER
Sep 29 19:01:08 dist-test-slave-lqkv krb5kdc[4456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708468, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 19:01:08.230178 4536 sys_catalog.cc:564] T 00000000000000000000000000000000 P ffcc85f05b4c427e9cd78e6fc2dbb71b [sys.catalog]: configured and running, proceeding with master startup.
I20260929 19:01:08.230221 4541 consensus_queue.cc:237] T 00000000000000000000000000000000 P ffcc85f05b4c427e9cd78e6fc2dbb71b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffcc85f05b4c427e9cd78e6fc2dbb71b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46623 } }
I20260929 19:01:08.230515 4541 sys_catalog.cc:455] T 00000000000000000000000000000000 P ffcc85f05b4c427e9cd78e6fc2dbb71b [sys.catalog]: SysCatalogTable state changed. Reason: New leader ffcc85f05b4c427e9cd78e6fc2dbb71b. Latest consensus state: current_term: 1 leader_uuid: "ffcc85f05b4c427e9cd78e6fc2dbb71b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffcc85f05b4c427e9cd78e6fc2dbb71b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46623 } } }
I20260929 19:01:08.230587 4541 sys_catalog.cc:458] T 00000000000000000000000000000000 P ffcc85f05b4c427e9cd78e6fc2dbb71b [sys.catalog]: This master's current role is: LEADER
I20260929 19:01:08.230814 4546 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 19:01:08.230525 4543 sys_catalog.cc:455] T 00000000000000000000000000000000 P ffcc85f05b4c427e9cd78e6fc2dbb71b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ffcc85f05b4c427e9cd78e6fc2dbb71b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffcc85f05b4c427e9cd78e6fc2dbb71b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46623 } } }
I20260929 19:01:08.231091 4543 sys_catalog.cc:458] T 00000000000000000000000000000000 P ffcc85f05b4c427e9cd78e6fc2dbb71b [sys.catalog]: This master's current role is: LEADER
I20260929 19:01:08.231390 4546 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 19:01:08.232986 4546 catalog_manager.cc:1347] Generated new cluster ID: 7fe3744568b34595920bf678fa4ee7e5
I20260929 19:01:08.233042 4546 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 19:01:08.234112 4539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:08.223955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45096 (local address 127.0.98.62:46623)
0929 19:01:08.224123 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:08.224130 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:08.224153 (+ 23us) server_negotiation.cc:407] Connection header received
0929 19:01:08.224521 (+ 368us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:08.224527 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:08.224610 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:08.224723 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:01:08.225540 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:08.226532 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:08.227574 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:08.227937 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:08.228013 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:08.230684 (+ 2671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:01:08.230688 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:01:08.230688 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:01:08.230700 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:01:08.232667 (+ 1967us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:08.233039 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:08.233042 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:08.233042 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:08.233128 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:08.233359 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:08.233359 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:08.233360 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:08.233584 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:01:08.233718 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0929 19:01:08.233830 (+ 112us) server_negotiation.cc:299] Negotiation successful
0929 19:01:08.233957 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":93,"threads_started":1}
I20260929 19:01:08.244148 4546 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 19:01:08.244920 4546 catalog_manager.cc:1504] Loading token signing keys...
I20260929 19:01:08.251756 4546 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ffcc85f05b4c427e9cd78e6fc2dbb71b: Generated new TSK 0
I20260929 19:01:08.251916 4546 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708447806542-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708447806542-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260929 19:01:08.279911 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUbnLHE/build/release/bin/kudu
/tmp/dist-test-taskUbnLHE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708447806542-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708447806542-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708447806542-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708447806542-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46623
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708447806542-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708447806542-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708447806542-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708447806542-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:01:08.358758 4562 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:01:08.358944 4562 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:01:08.358968 4562 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:01:08.360100 4562 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:01:08.360139 4562 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:01:08.360190 4562 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 19:01:08.361670 4562 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708447806542-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708447806542-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708447806542-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708447806542-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46623
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4562
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708447806542-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 18:09:13 UTC on ccycloud.quasar-hpkalf.root.comops.site
build id 4680913
I20260929 19:01:08.361945 4562 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:01:08.362164 4562 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:01:08.366765 4570 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:01:08.366849 4568 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:01:08.366771 4567 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:01:08.366986 4562 server_base.cc:1034] running on GCE node
I20260929 19:01:08.367187 4562 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:01:08.367372 4562 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:01:08.368537 4562 hybrid_clock.cc:648] HybridClock initialized: now 1790708468368535 us; error 33 us; skew 500 ppm
Sep 29 19:01:08 dist-test-slave-lqkv krb5kdc[4456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708468, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:01:08.370865 4562 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 19:01:08.371178 4562 fs_manager.cc:359] Metadata directory not provided
I20260929 19:01:08.371224 4562 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:01:08.371304 4562 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:01:08.372179 4562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708447806542-392-0/minicluster-data/ts-0/data/instance:
uuid: "5f3e33c85ac0412e8cf6c01314b28cd9"
format_stamp: "Formatted at 2026-09-29 19:01:08 on dist-test-slave-lqkv"
server_key: "d611a9a843b98437294b46253d800472"
server_key_iv: "3cd1f885afa9d6e7c3049706fbb974fc"
server_key_version: "encryptionkey@0"
I20260929 19:01:08.372468 4562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708447806542-392-0/minicluster-data/ts-0/wal/instance:
uuid: "5f3e33c85ac0412e8cf6c01314b28cd9"
format_stamp: "Formatted at 2026-09-29 19:01:08 on dist-test-slave-lqkv"
server_key: "d611a9a843b98437294b46253d800472"
server_key_iv: "3cd1f885afa9d6e7c3049706fbb974fc"
server_key_version: "encryptionkey@0"
I20260929 19:01:08.373663 4562 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 19:01:08.374279 4576 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:08.374454 4562 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 19:01:08.374543 4562 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708447806542-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708447806542-392-0/minicluster-data/ts-0/wal
uuid: "5f3e33c85ac0412e8cf6c01314b28cd9"
format_stamp: "Formatted at 2026-09-29 19:01:08 on dist-test-slave-lqkv"
server_key: "d611a9a843b98437294b46253d800472"
server_key_iv: "3cd1f885afa9d6e7c3049706fbb974fc"
server_key_version: "encryptionkey@0"
I20260929 19:01:08.374593 4562 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708447806542-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708447806542-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708447806542-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:01:08.392645 4562 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:01:08.394264 4562 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:01:08.394364 4562 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:01:08.394591 4562 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:01:08.394819 4562 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:01:08.394847 4562 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:08.394867 4562 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:01:08.394879 4562 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:08.400183 4562 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43995
I20260929 19:01:08.400250 4689 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43995 every 8 connection(s)
I20260929 19:01:08.400511 4562 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708447806542-392-0/minicluster-data/ts-0/data/info.pb
I20260929 19:01:08.404732 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUbnLHE/build/release/bin/kudu as pid 4562
I20260929 19:01:08.404875 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708447806542-392-0/minicluster-data/ts-0/wal/instance
I20260929 19:01:08.405071 392 external_mini_cluster.cc:1428] Setting key fc3b83826993ae1d03616c0f17aa2e58
Sep 29 19:01:08 dist-test-slave-lqkv krb5kdc[4456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708468, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 19:01:08.409823 4692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:08.401170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46623 (local address 127.0.98.1:46837)
0929 19:01:08.401344 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:08.401375 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0929 19:01:08.401804 (+ 429us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:01:08.402004 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:01:08.402007 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:01:08.402235 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:01:08.402693 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:08.402699 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:08.403311 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:08.403312 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:08.404066 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:08.404068 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:08.404432 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:08.404433 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:08.404525 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:08.405053 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:01:08.405061 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:01:08.406805 (+ 1744us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:01:08.408474 (+ 1669us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:01:08.408480 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:01:08.408481 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:01:08.408679 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:01:08.408938 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:01:08.408939 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:01:08.408940 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:01:08.409042 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:01:08.409300 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:01:08.409307 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:01:08.409428 (+ 121us) client_negotiation.cc:769] Sending connection context
0929 19:01:08.409560 (+ 132us) client_negotiation.cc:240] Negotiation successful
0929 19:01:08.409679 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":96,"threads_started":1}
I20260929 19:01:08.409825 4539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:08.401215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46837 (local address 127.0.98.62:46623)
0929 19:01:08.401287 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:08.401289 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:08.401450 (+ 161us) server_negotiation.cc:407] Connection header received
0929 19:01:08.401882 (+ 432us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:08.401883 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:08.401893 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:08.401948 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:01:08.402764 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:08.403238 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:08.404134 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:08.404376 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:08.404425 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:08.406888 (+ 2463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:01:08.406889 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:01:08.406889 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:01:08.406895 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:01:08.408405 (+ 1510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:08.408762 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:08.408763 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:08.408764 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:08.408871 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:08.409109 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:08.409110 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:08.409111 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:08.409237 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:01:08.409311 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 19:01:08.409618 (+ 307us) server_negotiation.cc:299] Negotiation successful
0929 19:01:08.409707 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 19:01:08.410388 4690 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46623
I20260929 19:01:08.410478 4690 heartbeater.cc:464] Registering TS with master...
I20260929 19:01:08.410629 4690 heartbeater.cc:511] Master 127.0.98.62:46623 requested a full tablet report, sending...
I20260929 19:01:08.411095 4500 ts_manager.cc:194] Registered new tserver with Master: 5f3e33c85ac0412e8cf6c01314b28cd9 (127.0.98.1:43995)
I20260929 19:01:08.411684 4500 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:46837
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708447806542-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708447806542-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260929 19:01:08.429304 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUbnLHE/build/release/bin/kudu
/tmp/dist-test-taskUbnLHE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708447806542-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708447806542-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708447806542-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708447806542-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46623
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708447806542-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708447806542-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708447806542-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708447806542-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:01:08.507689 4695 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:01:08.507912 4695 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:01:08.507951 4695 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:01:08.509106 4695 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:01:08.509171 4695 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:01:08.509228 4695 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 19:01:08.510700 4695 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708447806542-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708447806542-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708447806542-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708447806542-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46623
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4695
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708447806542-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 18:09:13 UTC on ccycloud.quasar-hpkalf.root.comops.site
build id 4680913
I20260929 19:01:08.510977 4695 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:01:08.511193 4695 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:01:08.515867 4701 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:01:08.515894 4703 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:01:08.515887 4700 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:01:08.516139 4695 server_base.cc:1034] running on GCE node
I20260929 19:01:08.516335 4695 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:01:08.516515 4695 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:01:08.517644 4695 hybrid_clock.cc:648] HybridClock initialized: now 1790708468517634 us; error 26 us; skew 500 ppm
Sep 29 19:01:08 dist-test-slave-lqkv krb5kdc[4456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708468, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:01:08.519896 4695 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 19:01:08.520233 4695 fs_manager.cc:359] Metadata directory not provided
I20260929 19:01:08.520315 4695 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:01:08.520465 4695 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:01:08.521528 4695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708447806542-392-0/minicluster-data/ts-1/data/instance:
uuid: "c59517b53cd141c599c8765865cb266a"
format_stamp: "Formatted at 2026-09-29 19:01:08 on dist-test-slave-lqkv"
server_key: "73ba26c6d75cee4260d9dfca58174a4b"
server_key_iv: "086a8e472a9dc6afecf0a820f8d2e224"
server_key_version: "encryptionkey@0"
I20260929 19:01:08.521871 4695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708447806542-392-0/minicluster-data/ts-1/wal/instance:
uuid: "c59517b53cd141c599c8765865cb266a"
format_stamp: "Formatted at 2026-09-29 19:01:08 on dist-test-slave-lqkv"
server_key: "73ba26c6d75cee4260d9dfca58174a4b"
server_key_iv: "086a8e472a9dc6afecf0a820f8d2e224"
server_key_version: "encryptionkey@0"
I20260929 19:01:08.523108 4695 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260929 19:01:08.523764 4709 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:08.523934 4695 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 19:01:08.524029 4695 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708447806542-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708447806542-392-0/minicluster-data/ts-1/wal
uuid: "c59517b53cd141c599c8765865cb266a"
format_stamp: "Formatted at 2026-09-29 19:01:08 on dist-test-slave-lqkv"
server_key: "73ba26c6d75cee4260d9dfca58174a4b"
server_key_iv: "086a8e472a9dc6afecf0a820f8d2e224"
server_key_version: "encryptionkey@0"
I20260929 19:01:08.524103 4695 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708447806542-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708447806542-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708447806542-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:01:08.544315 4695 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:01:08.546068 4695 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:01:08.546216 4695 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:01:08.546442 4695 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:01:08.546751 4695 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:01:08.546804 4695 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:08.546834 4695 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:01:08.546861 4695 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:08.552381 4695 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44873
I20260929 19:01:08.552446 4822 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44873 every 8 connection(s)
I20260929 19:01:08.552704 4695 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708447806542-392-0/minicluster-data/ts-1/data/info.pb
I20260929 19:01:08.553710 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUbnLHE/build/release/bin/kudu as pid 4695
I20260929 19:01:08.553802 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708447806542-392-0/minicluster-data/ts-1/wal/instance
I20260929 19:01:08.554001 392 external_mini_cluster.cc:1428] Setting key 59900cecfd76c4684af3f5e0723d6061
Sep 29 19:01:08 dist-test-slave-lqkv krb5kdc[4456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708468, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260929 19:01:08.562533 4539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:08.553384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58623 (local address 127.0.98.62:46623)
0929 19:01:08.553462 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:08.553464 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:08.553669 (+ 205us) server_negotiation.cc:407] Connection header received
0929 19:01:08.554319 (+ 650us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:08.554320 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:08.554331 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:08.554383 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:01:08.555575 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:08.556048 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:08.557087 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:08.557292 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:08.557362 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:08.559716 (+ 2354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:01:08.559718 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:01:08.559719 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:01:08.559726 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:01:08.561173 (+ 1447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:08.561575 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:08.561576 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:08.561577 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:08.561634 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:08.561905 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:08.561906 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:08.561906 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:08.562018 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:01:08.562092 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 19:01:08.562354 (+ 262us) server_negotiation.cc:299] Negotiation successful
0929 19:01:08.562431 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260929 19:01:08.562696 4825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:08.553357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46623 (local address 127.0.98.2:58623)
0929 19:01:08.553522 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:08.553574 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 19:01:08.554182 (+ 608us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:01:08.554440 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:01:08.554443 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:01:08.554828 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:01:08.555485 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:08.555491 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:08.556146 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:08.556147 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:08.556995 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:08.556997 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:08.557376 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:08.557377 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:08.557521 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:08.558043 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:01:08.558060 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:01:08.559613 (+ 1553us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:01:08.561268 (+ 1655us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:01:08.561274 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:01:08.561276 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:01:08.561487 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:01:08.561699 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:01:08.561700 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:01:08.561701 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:01:08.561852 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:01:08.562084 (+ 232us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:01:08.562089 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:01:08.562231 (+ 142us) client_negotiation.cc:769] Sending connection context
0929 19:01:08.562334 (+ 103us) client_negotiation.cc:240] Negotiation successful
0929 19:01:08.562477 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":77,"threads_started":1}
I20260929 19:01:08.563198 4823 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46623
I20260929 19:01:08.563271 4823 heartbeater.cc:464] Registering TS with master...
I20260929 19:01:08.563460 4823 heartbeater.cc:511] Master 127.0.98.62:46623 requested a full tablet report, sending...
I20260929 19:01:08.563860 4500 ts_manager.cc:194] Registered new tserver with Master: c59517b53cd141c599c8765865cb266a (127.0.98.2:44873)
I20260929 19:01:08.564254 4500 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:58623
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708447806542-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708447806542-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260929 19:01:08.577840 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUbnLHE/build/release/bin/kudu
/tmp/dist-test-taskUbnLHE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708447806542-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708447806542-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708447806542-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708447806542-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46623
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708447806542-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708447806542-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708447806542-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708447806542-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:01:08.655952 4828 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:01:08.656159 4828 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:01:08.656196 4828 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:01:08.657356 4828 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:01:08.657430 4828 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:01:08.657475 4828 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 19:01:08.658968 4828 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708447806542-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708447806542-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708447806542-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708447806542-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46623
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4828
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708447806542-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 18:09:13 UTC on ccycloud.quasar-hpkalf.root.comops.site
build id 4680913
I20260929 19:01:08.659212 4828 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:01:08.659442 4828 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:01:08.664187 4836 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:01:08.664186 4834 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:01:08.664414 4828 server_base.cc:1034] running on GCE node
W20260929 19:01:08.664227 4833 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:01:08.664677 4828 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:01:08.664968 4828 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:01:08.666129 4828 hybrid_clock.cc:648] HybridClock initialized: now 1790708468666129 us; error 40 us; skew 500 ppm
Sep 29 19:01:08 dist-test-slave-lqkv krb5kdc[4456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708468, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:01:08.668663 4828 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 19:01:08.669095 4828 fs_manager.cc:359] Metadata directory not provided
I20260929 19:01:08.669144 4828 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:01:08.669224 4828 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:01:08.670174 4828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708447806542-392-0/minicluster-data/ts-2/data/instance:
uuid: "60ac1f47bc2447338e8d68493f3214fc"
format_stamp: "Formatted at 2026-09-29 19:01:08 on dist-test-slave-lqkv"
server_key: "bba9673ee418d51154999fc6d0624cb8"
server_key_iv: "913108c41bcb6d0c4cded53afa4c78b6"
server_key_version: "encryptionkey@0"
I20260929 19:01:08.670464 4828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708447806542-392-0/minicluster-data/ts-2/wal/instance:
uuid: "60ac1f47bc2447338e8d68493f3214fc"
format_stamp: "Formatted at 2026-09-29 19:01:08 on dist-test-slave-lqkv"
server_key: "bba9673ee418d51154999fc6d0624cb8"
server_key_iv: "913108c41bcb6d0c4cded53afa4c78b6"
server_key_version: "encryptionkey@0"
I20260929 19:01:08.671609 4828 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260929 19:01:08.672370 4842 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:08.672546 4828 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 19:01:08.672631 4828 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708447806542-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708447806542-392-0/minicluster-data/ts-2/wal
uuid: "60ac1f47bc2447338e8d68493f3214fc"
format_stamp: "Formatted at 2026-09-29 19:01:08 on dist-test-slave-lqkv"
server_key: "bba9673ee418d51154999fc6d0624cb8"
server_key_iv: "913108c41bcb6d0c4cded53afa4c78b6"
server_key_version: "encryptionkey@0"
I20260929 19:01:08.672699 4828 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708447806542-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708447806542-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708447806542-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:01:08.685586 4828 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:01:08.687739 4828 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:01:08.687891 4828 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:01:08.688102 4828 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:01:08.688380 4828 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:01:08.688412 4828 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:08.688431 4828 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:01:08.688478 4828 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:08.694067 4828 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36753
I20260929 19:01:08.694108 4955 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36753 every 8 connection(s)
I20260929 19:01:08.694334 4828 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708447806542-392-0/minicluster-data/ts-2/data/info.pb
Sep 29 19:01:08 dist-test-slave-lqkv krb5kdc[4456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708468, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 19:01:08.702149 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUbnLHE/build/release/bin/kudu as pid 4828
I20260929 19:01:08.702260 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708447806542-392-0/minicluster-data/ts-2/wal/instance
I20260929 19:01:08.702452 392 external_mini_cluster.cc:1428] Setting key 91834d14ce32ff3b7eb3b5ecfa486692
I20260929 19:01:08.702761 4539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:08.694978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40779 (local address 127.0.98.62:46623)
0929 19:01:08.695046 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:08.695048 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:08.695224 (+ 176us) server_negotiation.cc:407] Connection header received
0929 19:01:08.695635 (+ 411us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:08.695636 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:08.695646 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:08.695702 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:01:08.696569 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:08.697137 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:08.698115 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:08.698299 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:08.698358 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:08.700247 (+ 1889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:01:08.700248 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:01:08.700248 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:01:08.700254 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:01:08.701393 (+ 1139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:08.701734 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:08.701735 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:08.701736 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:08.701791 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:08.702060 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:08.702061 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:08.702061 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:08.702187 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:01:08.702272 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0929 19:01:08.702601 (+ 329us) server_negotiation.cc:299] Negotiation successful
0929 19:01:08.702673 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 19:01:08.702929 4958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:08.694928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46623 (local address 127.0.98.3:40779)
0929 19:01:08.695086 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:08.695123 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 19:01:08.695551 (+ 428us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:01:08.695775 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:01:08.695779 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:01:08.696019 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:01:08.696449 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:08.696453 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:08.697218 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:08.697219 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:08.698026 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:08.698028 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:08.698378 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:08.698379 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:08.698489 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:08.698883 (+ 394us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:01:08.698891 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:01:08.700158 (+ 1267us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:01:08.701458 (+ 1300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:01:08.701463 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:01:08.701465 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:01:08.701657 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:01:08.701855 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:01:08.701856 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:01:08.701856 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:01:08.701939 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:01:08.702275 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:01:08.702281 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:01:08.702428 (+ 147us) client_negotiation.cc:769] Sending connection context
0929 19:01:08.702564 (+ 136us) client_negotiation.cc:240] Negotiation successful
0929 19:01:08.702710 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":89,"threads_started":1}
I20260929 19:01:08.703372 4956 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46623
I20260929 19:01:08.703469 4956 heartbeater.cc:464] Registering TS with master...
I20260929 19:01:08.703639 4956 heartbeater.cc:511] Master 127.0.98.62:46623 requested a full tablet report, sending...
I20260929 19:01:08.703989 4499 ts_manager.cc:194] Registered new tserver with Master: 60ac1f47bc2447338e8d68493f3214fc (127.0.98.3:36753)
I20260929 19:01:08.704319 4499 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:40779
I20260929 19:01:08.714296 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 19:01:08.720680 4539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:08.715498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41656 (local address 127.0.98.62:46623)
0929 19:01:08.715550 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:08.715551 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:08.715659 (+ 108us) server_negotiation.cc:407] Connection header received
0929 19:01:08.715696 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:08.715696 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:08.715704 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:08.715751 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:01:08.716278 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:08.716751 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:08.717470 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:08.717636 (+ 166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:08.717689 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:08.718813 (+ 1124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:01:08.718814 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:01:08.718814 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:01:08.718819 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:01:08.719877 (+ 1058us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:08.720154 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:08.720155 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:08.720155 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:08.720195 (+ 40us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:08.720328 (+ 133us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:08.720328 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:08.720329 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:08.720434 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:01:08.720496 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0929 19:01:08.720581 (+ 85us) server_negotiation.cc:299] Negotiation successful
0929 19:01:08.720629 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260929 19:01:08.722221 4499 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41656:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 19:01:08.723702 4499 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 19:01:08.730141 4966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:08.725776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32814 (local address 127.0.98.1:43995)
0929 19:01:08.726160 (+ 384us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:08.726165 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:08.726176 (+ 11us) server_negotiation.cc:407] Connection header received
0929 19:01:08.726215 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:08.726217 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:08.726239 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:08.726331 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:01:08.726770 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:08.727600 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:08.729186 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:08.729735 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:08.729817 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:08.729909 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0929 19:01:08.730014 (+ 105us) server_negotiation.cc:299] Negotiation successful
0929 19:01:08.730054 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":86,"threads_started":1}
I20260929 19:01:08.730255 4967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:08.725828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43962 (local address 127.0.98.2:44873)
0929 19:01:08.726030 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:08.726035 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:08.726101 (+ 66us) server_negotiation.cc:407] Connection header received
0929 19:01:08.726169 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:08.726171 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:08.726187 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:08.726521 (+ 334us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:01:08.726600 (+ 79us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:08.727721 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:08.729157 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:08.729815 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:08.730048 (+ 233us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:08.730106 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0929 19:01:08.730132 (+ 26us) server_negotiation.cc:299] Negotiation successful
0929 19:01:08.730171 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":121,"threads_started":1}
I20260929 19:01:08.730314 4969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:08.726068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43928 (local address 127.0.98.3:36753)
0929 19:01:08.726237 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:08.726240 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:08.726402 (+ 162us) server_negotiation.cc:407] Connection header received
0929 19:01:08.726449 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:08.726451 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:08.726467 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:08.726533 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:01:08.726718 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:08.727763 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:08.728960 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:08.729739 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:08.729807 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:08.729867 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0929 19:01:08.730181 (+ 314us) server_negotiation.cc:299] Negotiation successful
0929 19:01:08.730219 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":75,"threads_started":1}
I20260929 19:01:08.730145 4965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:08.725659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43995 (local address 127.0.0.1:32814)
0929 19:01:08.725866 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:08.725905 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 19:01:08.725967 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:01:08.726342 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:01:08.726347 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:01:08.726350 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:01:08.726503 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:08.726505 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:08.727744 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:08.727744 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:08.729103 (+ 1359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:08.729105 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:08.729828 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:08.729829 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:08.729915 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:08.729930 (+ 15us) client_negotiation.cc:769] Sending connection context
0929 19:01:08.730001 (+ 71us) client_negotiation.cc:240] Negotiation successful
0929 19:01:08.730059 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"mutex_wait_us":40,"thread_start_us":60,"threads_started":1}
I20260929 19:01:08.730335 4964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:08.725686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44873 (local address 127.0.0.1:43962)
0929 19:01:08.726058 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:08.726066 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0929 19:01:08.726108 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:01:08.726295 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:01:08.726356 (+ 61us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:01:08.726358 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:01:08.726444 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:08.726445 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:08.727821 (+ 1376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:08.727821 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:08.729071 (+ 1250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:08.729074 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:08.729901 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:08.729901 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:08.729970 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:08.729973 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 19:01:08.730002 (+ 29us) client_negotiation.cc:240] Negotiation successful
0929 19:01:08.730261 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":362,"mutex_wait_us":239,"thread_start_us":50,"threads_started":1}
I20260929 19:01:08.730149 4968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:08.725885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36753 (local address 127.0.0.1:43928)
0929 19:01:08.726281 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:08.726290 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 19:01:08.726334 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:01:08.726611 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:01:08.726612 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:01:08.726614 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:01:08.726650 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:08.726651 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:08.727860 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:08.727861 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:08.728872 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:08.728875 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:08.729899 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:08.729900 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:08.729960 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:08.729964 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 19:01:08.730013 (+ 49us) client_negotiation.cc:240] Negotiation successful
0929 19:01:08.730055 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":380,"mutex_wait_us":53,"thread_start_us":65,"threads_started":1}
I20260929 19:01:08.731400 4757 tablet_service.cc:1467] Processing CreateTablet for tablet d4d536f09d5f4431b2199dd406f9a2ac (DEFAULT_TABLE table=test-table [id=549ff2c21fba441aafc4018bbd1f420f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:01:08.731756 4757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4d536f09d5f4431b2199dd406f9a2ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:01:08.733187 4624 tablet_service.cc:1467] Processing CreateTablet for tablet d4d536f09d5f4431b2199dd406f9a2ac (DEFAULT_TABLE table=test-table [id=549ff2c21fba441aafc4018bbd1f420f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:01:08.733489 4624 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4d536f09d5f4431b2199dd406f9a2ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:01:08.734480 4970 tablet_bootstrap.cc:492] T d4d536f09d5f4431b2199dd406f9a2ac P c59517b53cd141c599c8765865cb266a: Bootstrap starting.
I20260929 19:01:08.735044 4970 tablet_bootstrap.cc:654] T d4d536f09d5f4431b2199dd406f9a2ac P c59517b53cd141c599c8765865cb266a: Neither blocks nor log segments found. Creating new log.
I20260929 19:01:08.735525 4970 log.cc:824] T d4d536f09d5f4431b2199dd406f9a2ac P c59517b53cd141c599c8765865cb266a: Log is configured to *not* fsync() on all Append() calls
I20260929 19:01:08.736208 4970 tablet_bootstrap.cc:492] T d4d536f09d5f4431b2199dd406f9a2ac P c59517b53cd141c599c8765865cb266a: No bootstrap required, opened a new log
I20260929 19:01:08.736290 4970 ts_tablet_manager.cc:1397] T d4d536f09d5f4431b2199dd406f9a2ac P c59517b53cd141c599c8765865cb266a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 19:01:08.736374 4972 tablet_bootstrap.cc:492] T d4d536f09d5f4431b2199dd406f9a2ac P 5f3e33c85ac0412e8cf6c01314b28cd9: Bootstrap starting.
I20260929 19:01:08.736676 4890 tablet_service.cc:1467] Processing CreateTablet for tablet d4d536f09d5f4431b2199dd406f9a2ac (DEFAULT_TABLE table=test-table [id=549ff2c21fba441aafc4018bbd1f420f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:01:08.736966 4972 tablet_bootstrap.cc:654] T d4d536f09d5f4431b2199dd406f9a2ac P 5f3e33c85ac0412e8cf6c01314b28cd9: Neither blocks nor log segments found. Creating new log.
I20260929 19:01:08.736974 4890 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4d536f09d5f4431b2199dd406f9a2ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:01:08.737520 4972 log.cc:824] T d4d536f09d5f4431b2199dd406f9a2ac P 5f3e33c85ac0412e8cf6c01314b28cd9: Log is configured to *not* fsync() on all Append() calls
I20260929 19:01:08.738265 4972 tablet_bootstrap.cc:492] T d4d536f09d5f4431b2199dd406f9a2ac P 5f3e33c85ac0412e8cf6c01314b28cd9: No bootstrap required, opened a new log
I20260929 19:01:08.738334 4972 ts_tablet_manager.cc:1397] T d4d536f09d5f4431b2199dd406f9a2ac P 5f3e33c85ac0412e8cf6c01314b28cd9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 19:01:08.738353 4970 raft_consensus.cc:348] T d4d536f09d5f4431b2199dd406f9a2ac P c59517b53cd141c599c8765865cb266a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f3e33c85ac0412e8cf6c01314b28cd9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43995 } } peers { permanent_uuid: "60ac1f47bc2447338e8d68493f3214fc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36753 } } peers { permanent_uuid: "c59517b53cd141c599c8765865cb266a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44873 } }
I20260929 19:01:08.738510 4970 raft_consensus.cc:374] T d4d536f09d5f4431b2199dd406f9a2ac P c59517b53cd141c599c8765865cb266a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:01:08.738545 4970 raft_consensus.cc:729] T d4d536f09d5f4431b2199dd406f9a2ac P c59517b53cd141c599c8765865cb266a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c59517b53cd141c599c8765865cb266a, State: Initialized, Role: FOLLOWER
I20260929 19:01:08.738658 4970 consensus_queue.cc:260] T d4d536f09d5f4431b2199dd406f9a2ac P c59517b53cd141c599c8765865cb266a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f3e33c85ac0412e8cf6c01314b28cd9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43995 } } peers { permanent_uuid: "60ac1f47bc2447338e8d68493f3214fc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36753 } } peers { permanent_uuid: "c59517b53cd141c599c8765865cb266a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44873 } }
I20260929 19:01:08.738965 4970 ts_tablet_manager.cc:1428] T d4d536f09d5f4431b2199dd406f9a2ac P c59517b53cd141c599c8765865cb266a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 19:01:08.739089 4823 heartbeater.cc:503] Master 127.0.98.62:46623 was elected leader, sending a full tablet report...
I20260929 19:01:08.740091 4975 tablet_bootstrap.cc:492] T d4d536f09d5f4431b2199dd406f9a2ac P 60ac1f47bc2447338e8d68493f3214fc: Bootstrap starting.
I20260929 19:01:08.740676 4975 tablet_bootstrap.cc:654] T d4d536f09d5f4431b2199dd406f9a2ac P 60ac1f47bc2447338e8d68493f3214fc: Neither blocks nor log segments found. Creating new log.
I20260929 19:01:08.740557 4972 raft_consensus.cc:348] T d4d536f09d5f4431b2199dd406f9a2ac P 5f3e33c85ac0412e8cf6c01314b28cd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f3e33c85ac0412e8cf6c01314b28cd9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43995 } } peers { permanent_uuid: "60ac1f47bc2447338e8d68493f3214fc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36753 } } peers { permanent_uuid: "c59517b53cd141c599c8765865cb266a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44873 } }
I20260929 19:01:08.740724 4972 raft_consensus.cc:374] T d4d536f09d5f4431b2199dd406f9a2ac P 5f3e33c85ac0412e8cf6c01314b28cd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:01:08.740759 4972 raft_consensus.cc:729] T d4d536f09d5f4431b2199dd406f9a2ac P 5f3e33c85ac0412e8cf6c01314b28cd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f3e33c85ac0412e8cf6c01314b28cd9, State: Initialized, Role: FOLLOWER
I20260929 19:01:08.740900 4972 consensus_queue.cc:260] T d4d536f09d5f4431b2199dd406f9a2ac P 5f3e33c85ac0412e8cf6c01314b28cd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f3e33c85ac0412e8cf6c01314b28cd9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43995 } } peers { permanent_uuid: "60ac1f47bc2447338e8d68493f3214fc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36753 } } peers { permanent_uuid: "c59517b53cd141c599c8765865cb266a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44873 } }
I20260929 19:01:08.741202 4972 ts_tablet_manager.cc:1428] T d4d536f09d5f4431b2199dd406f9a2ac P 5f3e33c85ac0412e8cf6c01314b28cd9: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 19:01:08.741207 4690 heartbeater.cc:503] Master 127.0.98.62:46623 was elected leader, sending a full tablet report...
I20260929 19:01:08.741289 4975 log.cc:824] T d4d536f09d5f4431b2199dd406f9a2ac P 60ac1f47bc2447338e8d68493f3214fc: Log is configured to *not* fsync() on all Append() calls
I20260929 19:01:08.742081 4975 tablet_bootstrap.cc:492] T d4d536f09d5f4431b2199dd406f9a2ac P 60ac1f47bc2447338e8d68493f3214fc: No bootstrap required, opened a new log
I20260929 19:01:08.742148 4975 ts_tablet_manager.cc:1397] T d4d536f09d5f4431b2199dd406f9a2ac P 60ac1f47bc2447338e8d68493f3214fc: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 19:01:08.743347 4975 raft_consensus.cc:348] T d4d536f09d5f4431b2199dd406f9a2ac P 60ac1f47bc2447338e8d68493f3214fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f3e33c85ac0412e8cf6c01314b28cd9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43995 } } peers { permanent_uuid: "60ac1f47bc2447338e8d68493f3214fc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36753 } } peers { permanent_uuid: "c59517b53cd141c599c8765865cb266a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44873 } }
I20260929 19:01:08.743453 4975 raft_consensus.cc:374] T d4d536f09d5f4431b2199dd406f9a2ac P 60ac1f47bc2447338e8d68493f3214fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:01:08.743474 4975 raft_consensus.cc:729] T d4d536f09d5f4431b2199dd406f9a2ac P 60ac1f47bc2447338e8d68493f3214fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60ac1f47bc2447338e8d68493f3214fc, State: Initialized, Role: FOLLOWER
I20260929 19:01:08.743594 4975 consensus_queue.cc:260] T d4d536f09d5f4431b2199dd406f9a2ac P 60ac1f47bc2447338e8d68493f3214fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f3e33c85ac0412e8cf6c01314b28cd9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43995 } } peers { permanent_uuid: "60ac1f47bc2447338e8d68493f3214fc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36753 } } peers { permanent_uuid: "c59517b53cd141c599c8765865cb266a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44873 } }
I20260929 19:01:08.743837 4975 ts_tablet_manager.cc:1428] T d4d536f09d5f4431b2199dd406f9a2ac P 60ac1f47bc2447338e8d68493f3214fc: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 19:01:08.743875 4956 heartbeater.cc:503] Master 127.0.98.62:46623 was elected leader, sending a full tablet report...
W20260929 19:01:08.803296 4824 tablet.cc:2369] T d4d536f09d5f4431b2199dd406f9a2ac P c59517b53cd141c599c8765865cb266a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 19:01:08.901285 4691 tablet.cc:2369] T d4d536f09d5f4431b2199dd406f9a2ac P 5f3e33c85ac0412e8cf6c01314b28cd9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 19:01:08.944892 4957 tablet.cc:2369] T d4d536f09d5f4431b2199dd406f9a2ac P 60ac1f47bc2447338e8d68493f3214fc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 19:01:09.118940 4978 raft_consensus.cc:482] T d4d536f09d5f4431b2199dd406f9a2ac P 60ac1f47bc2447338e8d68493f3214fc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 19:01:09.119079 4978 raft_consensus.cc:504] T d4d536f09d5f4431b2199dd406f9a2ac P 60ac1f47bc2447338e8d68493f3214fc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f3e33c85ac0412e8cf6c01314b28cd9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43995 } } peers { permanent_uuid: "60ac1f47bc2447338e8d68493f3214fc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36753 } } peers { permanent_uuid: "c59517b53cd141c599c8765865cb266a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44873 } }
I20260929 19:01:09.119521 4978 leader_election.cc:288] T d4d536f09d5f4431b2199dd406f9a2ac P 60ac1f47bc2447338e8d68493f3214fc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f3e33c85ac0412e8cf6c01314b28cd9 (127.0.98.1:43995), c59517b53cd141c599c8765865cb266a (127.0.98.2:44873)
I20260929 19:01:09.123452 4966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:09.119789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39351 (local address 127.0.98.1:43995)
0929 19:01:09.119862 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:09.119864 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:09.119875 (+ 11us) server_negotiation.cc:407] Connection header received
0929 19:01:09.119971 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:09.119972 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:09.119983 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:09.120037 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:01:09.120243 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:09.121040 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:09.122566 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:09.123094 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:09.123169 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:09.123178 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0929 19:01:09.123363 (+ 185us) server_negotiation.cc:299] Negotiation successful
0929 19:01:09.123396 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260929 19:01:09.123446 4967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:09.119883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39709 (local address 127.0.98.2:44873)
0929 19:01:09.119934 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:09.119936 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:09.120095 (+ 159us) server_negotiation.cc:407] Connection header received
0929 19:01:09.120129 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:09.120130 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:09.120139 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:09.120182 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:01:09.120349 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:09.121040 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:09.122566 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:09.123103 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:09.123169 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:09.123178 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0929 19:01:09.123335 (+ 157us) server_negotiation.cc:299] Negotiation successful
0929 19:01:09.123377 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260929 19:01:09.123457 4958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:09.119730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43995 (local address 127.0.98.3:39351)
0929 19:01:09.119806 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:09.119819 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 19:01:09.119861 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:01:09.120064 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:01:09.120065 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:01:09.120067 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:01:09.120140 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:09.120142 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:09.121126 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:09.121126 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:09.122461 (+ 1335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:09.122463 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:09.123177 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:09.123178 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:09.123260 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:09.123263 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 19:01:09.123310 (+ 47us) client_negotiation.cc:240] Negotiation successful
0929 19:01:09.123354 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54,"mutex_wait_us":123}
I20260929 19:01:09.123466 4979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:09.119758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44873 (local address 127.0.98.3:39709)
0929 19:01:09.119993 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:09.120006 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 19:01:09.120075 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:01:09.120216 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:01:09.120217 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:01:09.120218 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:01:09.120278 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:09.120279 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:09.121126 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:09.121126 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:09.122460 (+ 1334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:09.122463 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:09.123177 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:09.123178 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:09.123251 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:09.123255 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 19:01:09.123310 (+ 55us) client_negotiation.cc:240] Negotiation successful
0929 19:01:09.123354 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218,"mutex_wait_us":72,"thread_start_us":116,"threads_started":1}
I20260929 19:01:09.123862 4777 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4d536f09d5f4431b2199dd406f9a2ac" candidate_uuid: "60ac1f47bc2447338e8d68493f3214fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c59517b53cd141c599c8765865cb266a" is_pre_election: true
I20260929 19:01:09.123906 4644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4d536f09d5f4431b2199dd406f9a2ac" candidate_uuid: "60ac1f47bc2447338e8d68493f3214fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f3e33c85ac0412e8cf6c01314b28cd9" is_pre_election: true
I20260929 19:01:09.124007 4777 raft_consensus.cc:2454] T d4d536f09d5f4431b2199dd406f9a2ac P c59517b53cd141c599c8765865cb266a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60ac1f47bc2447338e8d68493f3214fc in term 0.
I20260929 19:01:09.124012 4644 raft_consensus.cc:2454] T d4d536f09d5f4431b2199dd406f9a2ac P 5f3e33c85ac0412e8cf6c01314b28cd9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60ac1f47bc2447338e8d68493f3214fc in term 0.
I20260929 19:01:09.124233 4843 leader_election.cc:302] T d4d536f09d5f4431b2199dd406f9a2ac P 60ac1f47bc2447338e8d68493f3214fc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60ac1f47bc2447338e8d68493f3214fc, c59517b53cd141c599c8765865cb266a; no voters:
I20260929 19:01:09.124365 4978 raft_consensus.cc:2781] T d4d536f09d5f4431b2199dd406f9a2ac P 60ac1f47bc2447338e8d68493f3214fc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 19:01:09.124435 4978 raft_consensus.cc:482] T d4d536f09d5f4431b2199dd406f9a2ac P 60ac1f47bc2447338e8d68493f3214fc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 19:01:09.124467 4978 raft_consensus.cc:3037] T d4d536f09d5f4431b2199dd406f9a2ac P 60ac1f47bc2447338e8d68493f3214fc [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:01:09.125169 4978 raft_consensus.cc:504] T d4d536f09d5f4431b2199dd406f9a2ac P 60ac1f47bc2447338e8d68493f3214fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f3e33c85ac0412e8cf6c01314b28cd9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43995 } } peers { permanent_uuid: "60ac1f47bc2447338e8d68493f3214fc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36753 } } peers { permanent_uuid: "c59517b53cd141c599c8765865cb266a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44873 } }
I20260929 19:01:09.125306 4978 leader_election.cc:288] T d4d536f09d5f4431b2199dd406f9a2ac P 60ac1f47bc2447338e8d68493f3214fc [CANDIDATE]: Term 1 election: Requested vote from peers 5f3e33c85ac0412e8cf6c01314b28cd9 (127.0.98.1:43995), c59517b53cd141c599c8765865cb266a (127.0.98.2:44873)
I20260929 19:01:09.125497 4777 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4d536f09d5f4431b2199dd406f9a2ac" candidate_uuid: "60ac1f47bc2447338e8d68493f3214fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c59517b53cd141c599c8765865cb266a"
I20260929 19:01:09.125497 4644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4d536f09d5f4431b2199dd406f9a2ac" candidate_uuid: "60ac1f47bc2447338e8d68493f3214fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f3e33c85ac0412e8cf6c01314b28cd9"
I20260929 19:01:09.125573 4644 raft_consensus.cc:3037] T d4d536f09d5f4431b2199dd406f9a2ac P 5f3e33c85ac0412e8cf6c01314b28cd9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:01:09.125574 4777 raft_consensus.cc:3037] T d4d536f09d5f4431b2199dd406f9a2ac P c59517b53cd141c599c8765865cb266a [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:01:09.126411 4644 raft_consensus.cc:2454] T d4d536f09d5f4431b2199dd406f9a2ac P 5f3e33c85ac0412e8cf6c01314b28cd9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60ac1f47bc2447338e8d68493f3214fc in term 1.
I20260929 19:01:09.126425 4777 raft_consensus.cc:2454] T d4d536f09d5f4431b2199dd406f9a2ac P c59517b53cd141c599c8765865cb266a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60ac1f47bc2447338e8d68493f3214fc in term 1.
I20260929 19:01:09.126578 4846 leader_election.cc:302] T d4d536f09d5f4431b2199dd406f9a2ac P 60ac1f47bc2447338e8d68493f3214fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f3e33c85ac0412e8cf6c01314b28cd9, 60ac1f47bc2447338e8d68493f3214fc; no voters:
I20260929 19:01:09.126681 4978 raft_consensus.cc:2781] T d4d536f09d5f4431b2199dd406f9a2ac P 60ac1f47bc2447338e8d68493f3214fc [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:01:09.126844 4978 raft_consensus.cc:686] T d4d536f09d5f4431b2199dd406f9a2ac P 60ac1f47bc2447338e8d68493f3214fc [term 1 LEADER]: Becoming Leader. State: Replica: 60ac1f47bc2447338e8d68493f3214fc, State: Running, Role: LEADER
I20260929 19:01:09.126938 4978 consensus_queue.cc:237] T d4d536f09d5f4431b2199dd406f9a2ac P 60ac1f47bc2447338e8d68493f3214fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f3e33c85ac0412e8cf6c01314b28cd9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43995 } } peers { permanent_uuid: "60ac1f47bc2447338e8d68493f3214fc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36753 } } peers { permanent_uuid: "c59517b53cd141c599c8765865cb266a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44873 } }
I20260929 19:01:09.127733 4498 catalog_manager.cc:5495] T d4d536f09d5f4431b2199dd406f9a2ac P 60ac1f47bc2447338e8d68493f3214fc reported cstate change: term changed from 0 to 1, leader changed from <none> to 60ac1f47bc2447338e8d68493f3214fc (127.0.98.3). New cstate: current_term: 1 leader_uuid: "60ac1f47bc2447338e8d68493f3214fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f3e33c85ac0412e8cf6c01314b28cd9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43995 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "60ac1f47bc2447338e8d68493f3214fc" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36753 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c59517b53cd141c599c8765865cb266a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44873 } health_report { overall_health: UNKNOWN } } }
I20260929 19:01:09.167505 4969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:09.165328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43930 (local address 127.0.98.3:36753)
0929 19:01:09.165406 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:09.165408 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:09.165416 (+ 8us) server_negotiation.cc:407] Connection header received
0929 19:01:09.165448 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:09.165448 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:09.165466 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:09.165527 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 19:01:09.165653 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:09.166097 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:09.166809 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:09.166982 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:09.167031 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:09.167170 (+ 139us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 19:01:09.167262 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 19:01:09.167320 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0929 19:01:09.167353 (+ 33us) server_negotiation.cc:299] Negotiation successful
0929 19:01:09.167368 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 19:01:09.169209 4644 raft_consensus.cc:1261] T d4d536f09d5f4431b2199dd406f9a2ac P 5f3e33c85ac0412e8cf6c01314b28cd9 [term 1 FOLLOWER]: Refusing update from remote peer 60ac1f47bc2447338e8d68493f3214fc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 19:01:09.169242 4777 raft_consensus.cc:1261] T d4d536f09d5f4431b2199dd406f9a2ac P c59517b53cd141c599c8765865cb266a [term 1 FOLLOWER]: Refusing update from remote peer 60ac1f47bc2447338e8d68493f3214fc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 19:01:09.169570 4980 consensus_queue.cc:1035] T d4d536f09d5f4431b2199dd406f9a2ac P 60ac1f47bc2447338e8d68493f3214fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f3e33c85ac0412e8cf6c01314b28cd9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 19:01:09.169687 4978 consensus_queue.cc:1035] T d4d536f09d5f4431b2199dd406f9a2ac P 60ac1f47bc2447338e8d68493f3214fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "c59517b53cd141c599c8765865cb266a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 19:01:09.172158 4984 mvcc.cc:205] Tried to move back new op lower bound from 7334741889714634752 to 7334741889544568832. Current Snapshot: MvccSnapshot[applied={T|T < 7334741889714634752}]
I20260929 19:01:09.172335 4985 mvcc.cc:205] Tried to move back new op lower bound from 7334741889714634752 to 7334741889544568832. Current Snapshot: MvccSnapshot[applied={T|T < 7334741889714634752}]
I20260929 19:01:09.174029 4499 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41656:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 19:01:09.174113 4499 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41656:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 19:01:09.174788 4499 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ffcc85f05b4c427e9cd78e6fc2dbb71b: Sending DeleteTablet for 3 replicas of tablet d4d536f09d5f4431b2199dd406f9a2ac
I20260929 19:01:09.175115 4757 tablet_service.cc:1514] Processing DeleteTablet for tablet d4d536f09d5f4431b2199dd406f9a2ac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:01:09 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43962
I20260929 19:01:09.175180 4890 tablet_service.cc:1514] Processing DeleteTablet for tablet d4d536f09d5f4431b2199dd406f9a2ac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:01:09 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43928
I20260929 19:01:09.175163 4624 tablet_service.cc:1514] Processing DeleteTablet for tablet d4d536f09d5f4431b2199dd406f9a2ac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:01:09 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:32814
I20260929 19:01:09.175390 4994 tablet_replica.cc:330] T d4d536f09d5f4431b2199dd406f9a2ac P 60ac1f47bc2447338e8d68493f3214fc: stopping tablet replica
I20260929 19:01:09.175498 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUbnLHE/build/release/bin/kudu with pid 4562
I20260929 19:01:09.176178 4993 tablet_replica.cc:330] T d4d536f09d5f4431b2199dd406f9a2ac P c59517b53cd141c599c8765865cb266a: stopping tablet replica
I20260929 19:01:09.176275 4993 raft_consensus.cc:2229] T d4d536f09d5f4431b2199dd406f9a2ac P c59517b53cd141c599c8765865cb266a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 19:01:09.176374 4993 raft_consensus.cc:2258] T d4d536f09d5f4431b2199dd406f9a2ac P c59517b53cd141c599c8765865cb266a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 19:01:09.176769 4993 ts_tablet_manager.cc:1905] T d4d536f09d5f4431b2199dd406f9a2ac P c59517b53cd141c599c8765865cb266a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 19:01:09.178185 4993 ts_tablet_manager.cc:1918] T d4d536f09d5f4431b2199dd406f9a2ac P c59517b53cd141c599c8765865cb266a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 19:01:09.178231 4993 log.cc:1196] T d4d536f09d5f4431b2199dd406f9a2ac P c59517b53cd141c599c8765865cb266a: Deleting WAL directory at /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708447806542-392-0/minicluster-data/ts-1/wal/wals/d4d536f09d5f4431b2199dd406f9a2ac
I20260929 19:01:09.178399 4993 ts_tablet_manager.cc:1939] T d4d536f09d5f4431b2199dd406f9a2ac P c59517b53cd141c599c8765865cb266a: Deleting consensus metadata
I20260929 19:01:09.178738 4484 catalog_manager.cc:4841] TS c59517b53cd141c599c8765865cb266a (127.0.98.2:44873): tablet d4d536f09d5f4431b2199dd406f9a2ac (table test-table [id=549ff2c21fba441aafc4018bbd1f420f]) successfully deleted
I20260929 19:01:09.178786 4994 raft_consensus.cc:2229] T d4d536f09d5f4431b2199dd406f9a2ac P 60ac1f47bc2447338e8d68493f3214fc [term 1 LEADER]: Raft consensus shutting down.
I20260929 19:01:09.178946 4994 raft_consensus.cc:2258] T d4d536f09d5f4431b2199dd406f9a2ac P 60ac1f47bc2447338e8d68493f3214fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 19:01:09.179261 4994 ts_tablet_manager.cc:1905] T d4d536f09d5f4431b2199dd406f9a2ac P 60ac1f47bc2447338e8d68493f3214fc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 19:01:09.180505 4994 ts_tablet_manager.cc:1918] T d4d536f09d5f4431b2199dd406f9a2ac P 60ac1f47bc2447338e8d68493f3214fc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 19:01:09.180583 4994 log.cc:1196] T d4d536f09d5f4431b2199dd406f9a2ac P 60ac1f47bc2447338e8d68493f3214fc: Deleting WAL directory at /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790708447806542-392-0/minicluster-data/ts-2/wal/wals/d4d536f09d5f4431b2199dd406f9a2ac
I20260929 19:01:09.180794 4994 ts_tablet_manager.cc:1939] T d4d536f09d5f4431b2199dd406f9a2ac P 60ac1f47bc2447338e8d68493f3214fc: Deleting consensus metadata
I20260929 19:01:09.181205 4487 catalog_manager.cc:4841] TS 60ac1f47bc2447338e8d68493f3214fc (127.0.98.3:36753): tablet d4d536f09d5f4431b2199dd406f9a2ac (table test-table [id=549ff2c21fba441aafc4018bbd1f420f]) successfully deleted
W20260929 19:01:09.182771 4487 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:43995 (error 108)
I20260929 19:01:09.183171 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUbnLHE/build/release/bin/kudu with pid 4695
I20260929 19:01:09.183229 4968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:09.183092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43995 (local address 127.0.0.1:32818)
0929 19:01:09.183152 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:09.183194 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260929 19:01:09.184262 4487 catalog_manager.cc:4584] TS 5f3e33c85ac0412e8cf6c01314b28cd9 (127.0.98.1:43995): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d4d536f09d5f4431b2199dd406f9a2ac: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43995: connect: Connection refused (error 111)
I20260929 19:01:09.188862 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUbnLHE/build/release/bin/kudu with pid 4828
I20260929 19:01:09.192627 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUbnLHE/build/release/bin/kudu with pid 4469
2026-09-29T19:01:09Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2229 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-29T19:01:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T19:01:09Z Disabled control of system clock
I20260929 19:01:09.219348 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUbnLHE/build/release/bin/kudu
/tmp/dist-test-taskUbnLHE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708447806542-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708447806542-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708447806542-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708447806542-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41945
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:45301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41945
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260929 19:01:09.295683 5000 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:01:09.295858 5000 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:01:09.295878 5000 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:01:09.296908 5000 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 19:01:09.296940 5000 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:01:09.296952 5000 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:01:09.296965 5000 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 19:01:09.296976 5000 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 19:01:09.298187 5000 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708447806542-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708447806542-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41945
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41945
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708447806542-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5000
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708447806542-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 18:09:13 UTC on ccycloud.quasar-hpkalf.root.comops.site
build id 4680913
I20260929 19:01:09.298362 5000 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:01:09.298584 5000 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:01:09.303191 5008 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:01:09.303205 5006 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:01:09.303205 5005 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:01:09.303256 5000 server_base.cc:1034] running on GCE node
I20260929 19:01:09.303617 5000 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:01:09.303879 5000 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:01:09.305015 5000 hybrid_clock.cc:648] HybridClock initialized: now 1790708469305001 us; error 32 us; skew 500 ppm
I20260929 19:01:09.306224 5000 webserver.cc:460] Webserver started at http://127.0.98.62:33195/ using document root <none> and password file <none>
I20260929 19:01:09.306557 5000 fs_manager.cc:359] Metadata directory not provided
I20260929 19:01:09.306607 5000 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:01:09.306751 5000 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:01:09.307756 5000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708447806542-392-0/minicluster-data/master-0/data/instance:
uuid: "16e8738dab0e402ca5f53d9b41c1c57c"
format_stamp: "Formatted at 2026-09-29 19:01:09 on dist-test-slave-lqkv"
server_key: "544603a3632720c3c93cefc8a925aec4"
server_key_iv: "2b78a2ff562bec647a986a6e92e0faab"
server_key_version: "encryptionkey@0"
I20260929 19:01:09.308111 5000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708447806542-392-0/minicluster-data/master-0/wal/instance:
uuid: "16e8738dab0e402ca5f53d9b41c1c57c"
format_stamp: "Formatted at 2026-09-29 19:01:09 on dist-test-slave-lqkv"
server_key: "544603a3632720c3c93cefc8a925aec4"
server_key_iv: "2b78a2ff562bec647a986a6e92e0faab"
server_key_version: "encryptionkey@0"
I20260929 19:01:09.309484 5000 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 19:01:09.310166 5014 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:09.310328 5000 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 19:01:09.310420 5000 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708447806542-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708447806542-392-0/minicluster-data/master-0/wal
uuid: "16e8738dab0e402ca5f53d9b41c1c57c"
format_stamp: "Formatted at 2026-09-29 19:01:09 on dist-test-slave-lqkv"
server_key: "544603a3632720c3c93cefc8a925aec4"
server_key_iv: "2b78a2ff562bec647a986a6e92e0faab"
server_key_version: "encryptionkey@0"
I20260929 19:01:09.310497 5000 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708447806542-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708447806542-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708447806542-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:01:09.313829 5000 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:01:09.314069 5000 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:01:09.314210 5000 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:01:09.317654 5000 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41945
I20260929 19:01:09.317684 5066 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41945 every 8 connection(s)
I20260929 19:01:09.317966 5000 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708447806542-392-0/minicluster-data/master-0/data/info.pb
I20260929 19:01:09.318569 5067 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:01:09.321087 5067 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16e8738dab0e402ca5f53d9b41c1c57c: Bootstrap starting.
I20260929 19:01:09.321594 5067 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 16e8738dab0e402ca5f53d9b41c1c57c: Neither blocks nor log segments found. Creating new log.
I20260929 19:01:09.321877 5067 log.cc:824] T 00000000000000000000000000000000 P 16e8738dab0e402ca5f53d9b41c1c57c: Log is configured to *not* fsync() on all Append() calls
I20260929 19:01:09.322470 5067 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16e8738dab0e402ca5f53d9b41c1c57c: No bootstrap required, opened a new log
I20260929 19:01:09.323236 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUbnLHE/build/release/bin/kudu as pid 5000
I20260929 19:01:09.323313 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708447806542-392-0/minicluster-data/master-0/wal/instance
I20260929 19:01:09.323482 392 external_mini_cluster.cc:1428] Setting key 7e6c2989490d0ae9e316c5e2830f84ee
I20260929 19:01:09.324482 5067 raft_consensus.cc:348] T 00000000000000000000000000000000 P 16e8738dab0e402ca5f53d9b41c1c57c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16e8738dab0e402ca5f53d9b41c1c57c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41945 } }
I20260929 19:01:09.324635 5067 raft_consensus.cc:374] T 00000000000000000000000000000000 P 16e8738dab0e402ca5f53d9b41c1c57c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:01:09.324689 5067 raft_consensus.cc:729] T 00000000000000000000000000000000 P 16e8738dab0e402ca5f53d9b41c1c57c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16e8738dab0e402ca5f53d9b41c1c57c, State: Initialized, Role: FOLLOWER
I20260929 19:01:09.324826 5067 consensus_queue.cc:260] T 00000000000000000000000000000000 P 16e8738dab0e402ca5f53d9b41c1c57c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16e8738dab0e402ca5f53d9b41c1c57c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41945 } }
I20260929 19:01:09.324930 5067 raft_consensus.cc:388] T 00000000000000000000000000000000 P 16e8738dab0e402ca5f53d9b41c1c57c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 19:01:09.324962 5067 raft_consensus.cc:482] T 00000000000000000000000000000000 P 16e8738dab0e402ca5f53d9b41c1c57c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 19:01:09.325001 5067 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 16e8738dab0e402ca5f53d9b41c1c57c [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:01:09.325532 5067 raft_consensus.cc:504] T 00000000000000000000000000000000 P 16e8738dab0e402ca5f53d9b41c1c57c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16e8738dab0e402ca5f53d9b41c1c57c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41945 } }
I20260929 19:01:09.325637 5067 leader_election.cc:302] T 00000000000000000000000000000000 P 16e8738dab0e402ca5f53d9b41c1c57c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 16e8738dab0e402ca5f53d9b41c1c57c; no voters:
I20260929 19:01:09.325793 5067 leader_election.cc:288] T 00000000000000000000000000000000 P 16e8738dab0e402ca5f53d9b41c1c57c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 19:01:09.325819 5072 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 16e8738dab0e402ca5f53d9b41c1c57c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:01:09.325944 5072 raft_consensus.cc:686] T 00000000000000000000000000000000 P 16e8738dab0e402ca5f53d9b41c1c57c [term 1 LEADER]: Becoming Leader. State: Replica: 16e8738dab0e402ca5f53d9b41c1c57c, State: Running, Role: LEADER
I20260929 19:01:09.326020 5072 consensus_queue.cc:237] T 00000000000000000000000000000000 P 16e8738dab0e402ca5f53d9b41c1c57c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16e8738dab0e402ca5f53d9b41c1c57c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41945 } }
I20260929 19:01:09.326046 5067 sys_catalog.cc:564] T 00000000000000000000000000000000 P 16e8738dab0e402ca5f53d9b41c1c57c [sys.catalog]: configured and running, proceeding with master startup.
I20260929 19:01:09.326135 5071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:09.323929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43524 (local address 127.0.98.62:41945)
0929 19:01:09.324141 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:09.324149 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:09.324180 (+ 31us) server_negotiation.cc:407] Connection header received
0929 19:01:09.324451 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:09.324459 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:09.324515 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:09.324682 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:01:09.325392 (+ 710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:01:09.325396 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:01:09.325396 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 19:01:09.325401 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:01:09.325446 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 19:01:09.325456 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:01:09.325532 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 19:01:09.325639 (+ 107us) server_negotiation.cc:299] Negotiation successful
0929 19:01:09.325694 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":84,"threads_started":1}
I20260929 19:01:09.326376 5073 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16e8738dab0e402ca5f53d9b41c1c57c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 16e8738dab0e402ca5f53d9b41c1c57c. Latest consensus state: current_term: 1 leader_uuid: "16e8738dab0e402ca5f53d9b41c1c57c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16e8738dab0e402ca5f53d9b41c1c57c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41945 } } }
I20260929 19:01:09.326601 5073 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16e8738dab0e402ca5f53d9b41c1c57c [sys.catalog]: This master's current role is: LEADER
I20260929 19:01:09.326545 5074 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16e8738dab0e402ca5f53d9b41c1c57c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "16e8738dab0e402ca5f53d9b41c1c57c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16e8738dab0e402ca5f53d9b41c1c57c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41945 } } }
I20260929 19:01:09.326859 5074 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16e8738dab0e402ca5f53d9b41c1c57c [sys.catalog]: This master's current role is: LEADER
I20260929 19:01:09.327018 5081 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 19:01:09.327528 5081 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 19:01:09.328987 5081 catalog_manager.cc:1347] Generated new cluster ID: 38674661ef6041bcb303578f8272d75e
I20260929 19:01:09.329056 5081 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 19:01:09.338665 5081 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 19:01:09.338809 5081 catalog_manager.cc:1504] Loading token signing keys...
I20260929 19:01:09.342571 5081 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 16e8738dab0e402ca5f53d9b41c1c57c: Generated new TSK 0
I20260929 19:01:09.342756 5081 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 19:01:09.349695 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUbnLHE/build/release/bin/kudu
/tmp/dist-test-taskUbnLHE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708447806542-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708447806542-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708447806542-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708447806542-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41945
--builtin_ntp_servers=127.0.98.20:45301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260929 19:01:09.434367 5091 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:01:09.434551 5091 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:01:09.434571 5091 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:01:09.435738 5091 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:01:09.435775 5091 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:01:09.435801 5091 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 19:01:09.437116 5091 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708447806542-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708447806542-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708447806542-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41945
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5091
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708447806542-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 18:09:13 UTC on ccycloud.quasar-hpkalf.root.comops.site
build id 4680913
I20260929 19:01:09.437323 5091 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:01:09.437620 5091 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:01:09.442968 5097 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:01:09.443094 5099 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:01:09.442982 5096 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:01:09.443014 5091 server_base.cc:1034] running on GCE node
I20260929 19:01:09.443320 5091 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:01:09.443567 5091 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:01:09.444729 5091 hybrid_clock.cc:648] HybridClock initialized: now 1790708469444702 us; error 25 us; skew 500 ppm
I20260929 19:01:09.446017 5091 webserver.cc:460] Webserver started at http://127.0.98.1:36987/ using document root <none> and password file <none>
I20260929 19:01:09.446237 5091 fs_manager.cc:359] Metadata directory not provided
I20260929 19:01:09.446280 5091 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:01:09.446421 5091 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:01:09.447396 5091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708447806542-392-0/minicluster-data/ts-0/data/instance:
uuid: "dc92236bb3cb43008381726c4f30b78f"
format_stamp: "Formatted at 2026-09-29 19:01:09 on dist-test-slave-lqkv"
server_key: "1940b26c4a90cfc52391353a0e1c3577"
server_key_iv: "9e5f976efa8cc265f5fb93ce95ac50ab"
server_key_version: "encryptionkey@0"
I20260929 19:01:09.447724 5091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708447806542-392-0/minicluster-data/ts-0/wal/instance:
uuid: "dc92236bb3cb43008381726c4f30b78f"
format_stamp: "Formatted at 2026-09-29 19:01:09 on dist-test-slave-lqkv"
server_key: "1940b26c4a90cfc52391353a0e1c3577"
server_key_iv: "9e5f976efa8cc265f5fb93ce95ac50ab"
server_key_version: "encryptionkey@0"
I20260929 19:01:09.449052 5091 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 19:01:09.449792 5105 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:09.449939 5091 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 19:01:09.450022 5091 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708447806542-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708447806542-392-0/minicluster-data/ts-0/wal
uuid: "dc92236bb3cb43008381726c4f30b78f"
format_stamp: "Formatted at 2026-09-29 19:01:09 on dist-test-slave-lqkv"
server_key: "1940b26c4a90cfc52391353a0e1c3577"
server_key_iv: "9e5f976efa8cc265f5fb93ce95ac50ab"
server_key_version: "encryptionkey@0"
I20260929 19:01:09.450089 5091 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708447806542-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708447806542-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708447806542-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:01:09.453269 5091 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:01:09.453521 5091 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:01:09.453657 5091 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:01:09.453900 5091 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:01:09.454210 5091 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:01:09.454264 5091 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:09.454301 5091 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:01:09.454326 5091 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:09.461556 5091 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46775
I20260929 19:01:09.461584 5218 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46775 every 8 connection(s)
I20260929 19:01:09.461926 5091 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708447806542-392-0/minicluster-data/ts-0/data/info.pb
I20260929 19:01:09.464918 5071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:09.462646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50727 (local address 127.0.98.62:41945)
0929 19:01:09.462709 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:09.462711 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:09.462955 (+ 244us) server_negotiation.cc:407] Connection header received
0929 19:01:09.463517 (+ 562us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:09.463518 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:09.463523 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:09.463568 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:01:09.464635 (+ 1067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:01:09.464636 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:01:09.464636 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 19:01:09.464640 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:01:09.464654 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 19:01:09.464658 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:01:09.464705 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0929 19:01:09.464790 (+ 85us) server_negotiation.cc:299] Negotiation successful
0929 19:01:09.464847 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260929 19:01:09.465106 5221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:09.462625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41945 (local address 127.0.98.1:50727)
0929 19:01:09.462810 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:09.462853 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 19:01:09.463432 (+ 579us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:01:09.463626 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:01:09.463629 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:01:09.463637 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:01:09.464504 (+ 867us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 19:01:09.464504 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:01:09.464530 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 19:01:09.464556 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:01:09.464716 (+ 160us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:01:09.464720 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:01:09.464722 (+ 2us) client_negotiation.cc:769] Sending connection context
0929 19:01:09.464777 (+ 55us) client_negotiation.cc:240] Negotiation successful
0929 19:01:09.464880 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":85,"threads_started":1}
I20260929 19:01:09.465332 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUbnLHE/build/release/bin/kudu as pid 5091
I20260929 19:01:09.465458 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708447806542-392-0/minicluster-data/ts-0/wal/instance
I20260929 19:01:09.465643 392 external_mini_cluster.cc:1428] Setting key 336a984660bae5ef09bb1f1024361f5d
I20260929 19:01:09.465641 5219 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41945
I20260929 19:01:09.465726 5219 heartbeater.cc:464] Registering TS with master...
I20260929 19:01:09.465881 5219 heartbeater.cc:511] Master 127.0.98.62:41945 requested a full tablet report, sending...
I20260929 19:01:09.466355 5031 ts_manager.cc:194] Registered new tserver with Master: dc92236bb3cb43008381726c4f30b78f (127.0.98.1:46775)
I20260929 19:01:09.466545 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUbnLHE/build/release/bin/kudu
/tmp/dist-test-taskUbnLHE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708447806542-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708447806542-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708447806542-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708447806542-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41945
--builtin_ntp_servers=127.0.98.20:45301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260929 19:01:09.543589 5222 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:01:09.543758 5222 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:01:09.543778 5222 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:01:09.544907 5222 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:01:09.544945 5222 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:01:09.544968 5222 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 19:01:09.546257 5222 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708447806542-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708447806542-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708447806542-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41945
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5222
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708447806542-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 18:09:13 UTC on ccycloud.quasar-hpkalf.root.comops.site
build id 4680913
I20260929 19:01:09.546509 5222 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:01:09.546770 5222 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:01:09.551132 5227 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:01:09.551137 5230 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:01:09.551367 5228 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:01:09.551514 5222 server_base.cc:1034] running on GCE node
I20260929 19:01:09.551733 5222 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:01:09.552006 5222 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:01:09.553149 5222 hybrid_clock.cc:648] HybridClock initialized: now 1790708469553138 us; error 23 us; skew 500 ppm
I20260929 19:01:09.554265 5222 webserver.cc:460] Webserver started at http://127.0.98.2:40293/ using document root <none> and password file <none>
I20260929 19:01:09.554513 5222 fs_manager.cc:359] Metadata directory not provided
I20260929 19:01:09.554595 5222 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:01:09.554723 5222 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:01:09.555694 5222 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708447806542-392-0/minicluster-data/ts-1/data/instance:
uuid: "384ef3c1fb364c709a580fd51de6feec"
format_stamp: "Formatted at 2026-09-29 19:01:09 on dist-test-slave-lqkv"
server_key: "ed7f4453669f2294c09e9df13e599663"
server_key_iv: "dc6c8a3d3b7ff32fd46db1c64bf0ae77"
server_key_version: "encryptionkey@0"
I20260929 19:01:09.556057 5222 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708447806542-392-0/minicluster-data/ts-1/wal/instance:
uuid: "384ef3c1fb364c709a580fd51de6feec"
format_stamp: "Formatted at 2026-09-29 19:01:09 on dist-test-slave-lqkv"
server_key: "ed7f4453669f2294c09e9df13e599663"
server_key_iv: "dc6c8a3d3b7ff32fd46db1c64bf0ae77"
server_key_version: "encryptionkey@0"
I20260929 19:01:09.557397 5222 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 19:01:09.558028 5236 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:09.558218 5222 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 19:01:09.558307 5222 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708447806542-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708447806542-392-0/minicluster-data/ts-1/wal
uuid: "384ef3c1fb364c709a580fd51de6feec"
format_stamp: "Formatted at 2026-09-29 19:01:09 on dist-test-slave-lqkv"
server_key: "ed7f4453669f2294c09e9df13e599663"
server_key_iv: "dc6c8a3d3b7ff32fd46db1c64bf0ae77"
server_key_version: "encryptionkey@0"
I20260929 19:01:09.558378 5222 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708447806542-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708447806542-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708447806542-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:01:09.561626 5222 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:01:09.561861 5222 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:01:09.561998 5222 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:01:09.562180 5222 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:01:09.562453 5222 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:01:09.562501 5222 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:09.562538 5222 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:01:09.562565 5222 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:09.567888 5222 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36699
I20260929 19:01:09.567940 5349 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36699 every 8 connection(s)
I20260929 19:01:09.568233 5222 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708447806542-392-0/minicluster-data/ts-1/data/info.pb
I20260929 19:01:09.570648 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUbnLHE/build/release/bin/kudu as pid 5222
I20260929 19:01:09.570763 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708447806542-392-0/minicluster-data/ts-1/wal/instance
I20260929 19:01:09.570930 392 external_mini_cluster.cc:1428] Setting key c7556e794cb508beeab4b7db1473bc49
I20260929 19:01:09.570950 5071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:09.568914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58149 (local address 127.0.98.62:41945)
0929 19:01:09.568970 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:09.568971 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:09.569206 (+ 235us) server_negotiation.cc:407] Connection header received
0929 19:01:09.569627 (+ 421us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:09.569628 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:09.569634 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:09.569687 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:01:09.570679 (+ 992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:01:09.570681 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:01:09.570681 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 19:01:09.570685 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:01:09.570701 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 19:01:09.570705 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:01:09.570756 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0929 19:01:09.570870 (+ 114us) server_negotiation.cc:299] Negotiation successful
0929 19:01:09.570885 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260929 19:01:09.571108 5352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:09.568924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41945 (local address 127.0.98.2:58149)
0929 19:01:09.569074 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:09.569125 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 19:01:09.569534 (+ 409us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:01:09.569750 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:01:09.569753 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:01:09.569762 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:01:09.570544 (+ 782us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 19:01:09.570545 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:01:09.570567 (+ 22us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 19:01:09.570591 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:01:09.570772 (+ 181us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:01:09.570776 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:01:09.570778 (+ 2us) client_negotiation.cc:769] Sending connection context
0929 19:01:09.570868 (+ 90us) client_negotiation.cc:240] Negotiation successful
0929 19:01:09.570921 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":73,"threads_started":1}
I20260929 19:01:09.571612 5350 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41945
I20260929 19:01:09.571696 5350 heartbeater.cc:464] Registering TS with master...
I20260929 19:01:09.571834 5350 heartbeater.cc:511] Master 127.0.98.62:41945 requested a full tablet report, sending...
I20260929 19:01:09.571890 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUbnLHE/build/release/bin/kudu
/tmp/dist-test-taskUbnLHE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708447806542-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708447806542-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708447806542-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708447806542-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41945
--builtin_ntp_servers=127.0.98.20:45301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260929 19:01:09.572299 5031 ts_manager.cc:194] Registered new tserver with Master: 384ef3c1fb364c709a580fd51de6feec (127.0.98.2:36699)
W20260929 19:01:09.650357 5353 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:01:09.650695 5353 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:01:09.650802 5353 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:01:09.652050 5353 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:01:09.652133 5353 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:01:09.652194 5353 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 19:01:09.653723 5353 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708447806542-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708447806542-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708447806542-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41945
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5353
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708447806542-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 18:09:13 UTC on ccycloud.quasar-hpkalf.root.comops.site
build id 4680913
I20260929 19:01:09.653975 5353 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:01:09.654241 5353 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:01:09.658769 5359 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:01:09.658778 5361 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:01:09.658780 5358 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:01:09.659196 5353 server_base.cc:1034] running on GCE node
I20260929 19:01:09.659432 5353 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:01:09.659646 5353 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:01:09.660792 5353 hybrid_clock.cc:648] HybridClock initialized: now 1790708469660790 us; error 33 us; skew 500 ppm
I20260929 19:01:09.662578 5353 webserver.cc:460] Webserver started at http://127.0.98.3:36131/ using document root <none> and password file <none>
I20260929 19:01:09.662846 5353 fs_manager.cc:359] Metadata directory not provided
I20260929 19:01:09.662911 5353 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:01:09.663035 5353 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:01:09.664060 5353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708447806542-392-0/minicluster-data/ts-2/data/instance:
uuid: "6d140f94dbbb429ea2f62e990effb1af"
format_stamp: "Formatted at 2026-09-29 19:01:09 on dist-test-slave-lqkv"
server_key: "9575c839c6125cbba7cdf0032669d47a"
server_key_iv: "fabbac23975720ec66dc667a5277b3c0"
server_key_version: "encryptionkey@0"
I20260929 19:01:09.664425 5353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708447806542-392-0/minicluster-data/ts-2/wal/instance:
uuid: "6d140f94dbbb429ea2f62e990effb1af"
format_stamp: "Formatted at 2026-09-29 19:01:09 on dist-test-slave-lqkv"
server_key: "9575c839c6125cbba7cdf0032669d47a"
server_key_iv: "fabbac23975720ec66dc667a5277b3c0"
server_key_version: "encryptionkey@0"
I20260929 19:01:09.665810 5353 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 19:01:09.666622 5367 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:09.666823 5353 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 19:01:09.666900 5353 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708447806542-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708447806542-392-0/minicluster-data/ts-2/wal
uuid: "6d140f94dbbb429ea2f62e990effb1af"
format_stamp: "Formatted at 2026-09-29 19:01:09 on dist-test-slave-lqkv"
server_key: "9575c839c6125cbba7cdf0032669d47a"
server_key_iv: "fabbac23975720ec66dc667a5277b3c0"
server_key_version: "encryptionkey@0"
I20260929 19:01:09.666973 5353 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708447806542-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708447806542-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708447806542-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:01:09.671329 5353 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:01:09.671591 5353 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:01:09.671711 5353 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:01:09.671919 5353 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:01:09.672200 5353 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:01:09.672252 5353 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:09.672290 5353 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:01:09.672327 5353 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:09.677842 5353 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33863
I20260929 19:01:09.677883 5480 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33863 every 8 connection(s)
I20260929 19:01:09.678197 5353 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708447806542-392-0/minicluster-data/ts-2/data/info.pb
I20260929 19:01:09.680783 5071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:09.678880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50767 (local address 127.0.98.62:41945)
0929 19:01:09.678936 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:09.678938 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:09.679166 (+ 228us) server_negotiation.cc:407] Connection header received
0929 19:01:09.679560 (+ 394us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:09.679561 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:09.679565 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:09.679619 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:01:09.680606 (+ 987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:01:09.680607 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:01:09.680608 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 19:01:09.680612 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:01:09.680626 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 19:01:09.680630 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:01:09.680675 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0929 19:01:09.680720 (+ 45us) server_negotiation.cc:299] Negotiation successful
0929 19:01:09.680732 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260929 19:01:09.681010 5483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:09.678878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41945 (local address 127.0.98.3:50767)
0929 19:01:09.679043 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:09.679077 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0929 19:01:09.679489 (+ 412us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:01:09.679663 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:01:09.679666 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:01:09.679673 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:01:09.680482 (+ 809us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 19:01:09.680482 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:01:09.680505 (+ 23us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 19:01:09.680529 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:01:09.680672 (+ 143us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:01:09.680674 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:01:09.680676 (+ 2us) client_negotiation.cc:769] Sending connection context
0929 19:01:09.680719 (+ 43us) client_negotiation.cc:240] Negotiation successful
0929 19:01:09.680759 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":83,"threads_started":1}
I20260929 19:01:09.681470 5481 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41945
I20260929 19:01:09.681566 5481 heartbeater.cc:464] Registering TS with master...
I20260929 19:01:09.681715 5481 heartbeater.cc:511] Master 127.0.98.62:41945 requested a full tablet report, sending...
I20260929 19:01:09.682039 5031 ts_manager.cc:194] Registered new tserver with Master: 6d140f94dbbb429ea2f62e990effb1af (127.0.98.3:33863)
I20260929 19:01:09.686070 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUbnLHE/build/release/bin/kudu as pid 5353
I20260929 19:01:09.686180 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708447806542-392-0/minicluster-data/ts-2/wal/instance
I20260929 19:01:09.686376 392 external_mini_cluster.cc:1428] Setting key bf5fe213ec3876918de7da290c43fe50
I20260929 19:01:09.687172 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 19:01:09.689435 5071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:09.688465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43540 (local address 127.0.98.62:41945)
0929 19:01:09.688537 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:09.688538 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:09.688589 (+ 51us) server_negotiation.cc:407] Connection header received
0929 19:01:09.688631 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:09.688632 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:09.688635 (+ 3us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:09.688678 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:01:09.689212 (+ 534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:01:09.689213 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:01:09.689214 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 19:01:09.689217 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:01:09.689229 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 19:01:09.689232 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:01:09.689278 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0929 19:01:09.689314 (+ 36us) server_negotiation.cc:299] Negotiation successful
0929 19:01:09.689325 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 19:01:09.690829 5031 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43540:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 19:01:09.692274 5031 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 19:01:09.695497 5489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:09.694143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33863 (local address 127.0.0.1:43506)
0929 19:01:09.694234 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:09.694286 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 19:01:09.694333 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:01:09.694667 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:01:09.694672 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:01:09.694678 (+ 6us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:01:09.695105 (+ 427us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 19:01:09.695106 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:01:09.695121 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 19:01:09.695142 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:01:09.695364 (+ 222us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:01:09.695369 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:01:09.695372 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 19:01:09.695416 (+ 44us) client_negotiation.cc:240] Negotiation successful
0929 19:01:09.695428 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73,"thread_start_us":52,"threads_started":1}
I20260929 19:01:09.695765 5491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:09.694337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43506 (local address 127.0.98.3:33863)
0929 19:01:09.694508 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:09.694512 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:09.694522 (+ 10us) server_negotiation.cc:407] Connection header received
0929 19:01:09.694543 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:09.694545 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:09.694553 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:09.694662 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:01:09.695225 (+ 563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:01:09.695230 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:01:09.695231 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 19:01:09.695235 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:01:09.695268 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 19:01:09.695286 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:01:09.695627 (+ 341us) server_negotiation.cc:1035] Waiting for connection context
0929 19:01:09.695660 (+ 33us) server_negotiation.cc:299] Negotiation successful
0929 19:01:09.695675 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":87,"threads_started":1}
I20260929 19:01:09.696091 5492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:09.694364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36699 (local address 127.0.0.1:49470)
0929 19:01:09.694504 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:09.694514 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 19:01:09.694549 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:01:09.695273 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:01:09.695274 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:01:09.695277 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:01:09.695686 (+ 409us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 19:01:09.695686 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:01:09.695692 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 19:01:09.695703 (+ 11us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:01:09.695971 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:01:09.695972 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:01:09.695973 (+ 1us) client_negotiation.cc:769] Sending connection context
0929 19:01:09.696016 (+ 43us) client_negotiation.cc:240] Negotiation successful
0929 19:01:09.696027 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":105,"threads_started":1}
I20260929 19:01:09.696286 5494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:09.694678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49470 (local address 127.0.98.2:36699)
0929 19:01:09.694861 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:09.694865 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:09.694875 (+ 10us) server_negotiation.cc:407] Connection header received
0929 19:01:09.694920 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:09.694922 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:09.694931 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:09.695003 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:01:09.695787 (+ 784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:01:09.695794 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:01:09.695794 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 19:01:09.695799 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:01:09.695834 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 19:01:09.695869 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:01:09.695945 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 19:01:09.696187 (+ 242us) server_negotiation.cc:299] Negotiation successful
0929 19:01:09.696205 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":81,"threads_started":1}
I20260929 19:01:09.696426 5490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:09.694179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46775 (local address 127.0.0.1:56480)
0929 19:01:09.694324 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:09.694334 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 19:01:09.694381 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:01:09.694804 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:01:09.694805 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:01:09.694808 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:01:09.695452 (+ 644us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 19:01:09.695453 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:01:09.695460 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 19:01:09.695472 (+ 12us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:01:09.696296 (+ 824us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:01:09.696297 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:01:09.696297 (+ 0us) client_negotiation.cc:769] Sending connection context
0929 19:01:09.696345 (+ 48us) client_negotiation.cc:240] Negotiation successful
0929 19:01:09.696359 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":100,"threads_started":1}
I20260929 19:01:09.696372 5415 tablet_service.cc:1467] Processing CreateTablet for tablet 6771d40d77af47b580939f3f15344912 (DEFAULT_TABLE table=test-table [id=306d5938d84d411ba63f079e55d7121b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:01:09.696609 5493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:09.694499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56480 (local address 127.0.98.1:46775)
0929 19:01:09.694656 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:09.694659 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:09.694674 (+ 15us) server_negotiation.cc:407] Connection header received
0929 19:01:09.694696 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:09.694698 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:09.694706 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:09.695369 (+ 663us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:01:09.696067 (+ 698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:01:09.696070 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:01:09.696070 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 19:01:09.696073 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:01:09.696098 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 19:01:09.696110 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:01:09.696301 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0929 19:01:09.696481 (+ 180us) server_negotiation.cc:299] Negotiation successful
0929 19:01:09.696515 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":84,"threads_started":1}
I20260929 19:01:09.696720 5415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6771d40d77af47b580939f3f15344912. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:01:09.697088 5284 tablet_service.cc:1467] Processing CreateTablet for tablet 6771d40d77af47b580939f3f15344912 (DEFAULT_TABLE table=test-table [id=306d5938d84d411ba63f079e55d7121b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:01:09.697199 5144 tablet_service.cc:1467] Processing CreateTablet for tablet 6771d40d77af47b580939f3f15344912 (DEFAULT_TABLE table=test-table [id=306d5938d84d411ba63f079e55d7121b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:01:09.697409 5284 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6771d40d77af47b580939f3f15344912. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:01:09.697525 5144 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6771d40d77af47b580939f3f15344912. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:01:09.699877 5495 tablet_bootstrap.cc:492] T 6771d40d77af47b580939f3f15344912 P 6d140f94dbbb429ea2f62e990effb1af: Bootstrap starting.
I20260929 19:01:09.700233 5496 tablet_bootstrap.cc:492] T 6771d40d77af47b580939f3f15344912 P dc92236bb3cb43008381726c4f30b78f: Bootstrap starting.
I20260929 19:01:09.700603 5495 tablet_bootstrap.cc:654] T 6771d40d77af47b580939f3f15344912 P 6d140f94dbbb429ea2f62e990effb1af: Neither blocks nor log segments found. Creating new log.
I20260929 19:01:09.700733 5497 tablet_bootstrap.cc:492] T 6771d40d77af47b580939f3f15344912 P 384ef3c1fb364c709a580fd51de6feec: Bootstrap starting.
I20260929 19:01:09.700920 5496 tablet_bootstrap.cc:654] T 6771d40d77af47b580939f3f15344912 P dc92236bb3cb43008381726c4f30b78f: Neither blocks nor log segments found. Creating new log.
I20260929 19:01:09.700949 5495 log.cc:824] T 6771d40d77af47b580939f3f15344912 P 6d140f94dbbb429ea2f62e990effb1af: Log is configured to *not* fsync() on all Append() calls
I20260929 19:01:09.701263 5496 log.cc:824] T 6771d40d77af47b580939f3f15344912 P dc92236bb3cb43008381726c4f30b78f: Log is configured to *not* fsync() on all Append() calls
I20260929 19:01:09.701478 5497 tablet_bootstrap.cc:654] T 6771d40d77af47b580939f3f15344912 P 384ef3c1fb364c709a580fd51de6feec: Neither blocks nor log segments found. Creating new log.
I20260929 19:01:09.701710 5495 tablet_bootstrap.cc:492] T 6771d40d77af47b580939f3f15344912 P 6d140f94dbbb429ea2f62e990effb1af: No bootstrap required, opened a new log
I20260929 19:01:09.701787 5495 ts_tablet_manager.cc:1397] T 6771d40d77af47b580939f3f15344912 P 6d140f94dbbb429ea2f62e990effb1af: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 19:01:09.701808 5497 log.cc:824] T 6771d40d77af47b580939f3f15344912 P 384ef3c1fb364c709a580fd51de6feec: Log is configured to *not* fsync() on all Append() calls
I20260929 19:01:09.702093 5496 tablet_bootstrap.cc:492] T 6771d40d77af47b580939f3f15344912 P dc92236bb3cb43008381726c4f30b78f: No bootstrap required, opened a new log
I20260929 19:01:09.702162 5496 ts_tablet_manager.cc:1397] T 6771d40d77af47b580939f3f15344912 P dc92236bb3cb43008381726c4f30b78f: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260929 19:01:09.702706 5497 tablet_bootstrap.cc:492] T 6771d40d77af47b580939f3f15344912 P 384ef3c1fb364c709a580fd51de6feec: No bootstrap required, opened a new log
I20260929 19:01:09.702790 5497 ts_tablet_manager.cc:1397] T 6771d40d77af47b580939f3f15344912 P 384ef3c1fb364c709a580fd51de6feec: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260929 19:01:09.704437 5495 raft_consensus.cc:348] T 6771d40d77af47b580939f3f15344912 P 6d140f94dbbb429ea2f62e990effb1af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d140f94dbbb429ea2f62e990effb1af" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33863 } } peers { permanent_uuid: "384ef3c1fb364c709a580fd51de6feec" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36699 } } peers { permanent_uuid: "dc92236bb3cb43008381726c4f30b78f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46775 } }
I20260929 19:01:09.704480 5497 raft_consensus.cc:348] T 6771d40d77af47b580939f3f15344912 P 384ef3c1fb364c709a580fd51de6feec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d140f94dbbb429ea2f62e990effb1af" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33863 } } peers { permanent_uuid: "384ef3c1fb364c709a580fd51de6feec" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36699 } } peers { permanent_uuid: "dc92236bb3cb43008381726c4f30b78f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46775 } }
I20260929 19:01:09.704595 5495 raft_consensus.cc:374] T 6771d40d77af47b580939f3f15344912 P 6d140f94dbbb429ea2f62e990effb1af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:01:09.704617 5497 raft_consensus.cc:374] T 6771d40d77af47b580939f3f15344912 P 384ef3c1fb364c709a580fd51de6feec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:01:09.704629 5495 raft_consensus.cc:729] T 6771d40d77af47b580939f3f15344912 P 6d140f94dbbb429ea2f62e990effb1af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d140f94dbbb429ea2f62e990effb1af, State: Initialized, Role: FOLLOWER
I20260929 19:01:09.704661 5497 raft_consensus.cc:729] T 6771d40d77af47b580939f3f15344912 P 384ef3c1fb364c709a580fd51de6feec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 384ef3c1fb364c709a580fd51de6feec, State: Initialized, Role: FOLLOWER
I20260929 19:01:09.704747 5495 consensus_queue.cc:260] T 6771d40d77af47b580939f3f15344912 P 6d140f94dbbb429ea2f62e990effb1af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d140f94dbbb429ea2f62e990effb1af" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33863 } } peers { permanent_uuid: "384ef3c1fb364c709a580fd51de6feec" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36699 } } peers { permanent_uuid: "dc92236bb3cb43008381726c4f30b78f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46775 } }
I20260929 19:01:09.704701 5496 raft_consensus.cc:348] T 6771d40d77af47b580939f3f15344912 P dc92236bb3cb43008381726c4f30b78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d140f94dbbb429ea2f62e990effb1af" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33863 } } peers { permanent_uuid: "384ef3c1fb364c709a580fd51de6feec" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36699 } } peers { permanent_uuid: "dc92236bb3cb43008381726c4f30b78f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46775 } }
I20260929 19:01:09.704854 5496 raft_consensus.cc:374] T 6771d40d77af47b580939f3f15344912 P dc92236bb3cb43008381726c4f30b78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:01:09.704841 5497 consensus_queue.cc:260] T 6771d40d77af47b580939f3f15344912 P 384ef3c1fb364c709a580fd51de6feec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d140f94dbbb429ea2f62e990effb1af" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33863 } } peers { permanent_uuid: "384ef3c1fb364c709a580fd51de6feec" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36699 } } peers { permanent_uuid: "dc92236bb3cb43008381726c4f30b78f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46775 } }
I20260929 19:01:09.704914 5496 raft_consensus.cc:729] T 6771d40d77af47b580939f3f15344912 P dc92236bb3cb43008381726c4f30b78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc92236bb3cb43008381726c4f30b78f, State: Initialized, Role: FOLLOWER
I20260929 19:01:09.705013 5495 ts_tablet_manager.cc:1428] T 6771d40d77af47b580939f3f15344912 P 6d140f94dbbb429ea2f62e990effb1af: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260929 19:01:09.705020 5496 consensus_queue.cc:260] T 6771d40d77af47b580939f3f15344912 P dc92236bb3cb43008381726c4f30b78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d140f94dbbb429ea2f62e990effb1af" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33863 } } peers { permanent_uuid: "384ef3c1fb364c709a580fd51de6feec" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36699 } } peers { permanent_uuid: "dc92236bb3cb43008381726c4f30b78f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46775 } }
I20260929 19:01:09.705091 5497 ts_tablet_manager.cc:1428] T 6771d40d77af47b580939f3f15344912 P 384ef3c1fb364c709a580fd51de6feec: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260929 19:01:09.705122 5481 heartbeater.cc:503] Master 127.0.98.62:41945 was elected leader, sending a full tablet report...
I20260929 19:01:09.705204 5350 heartbeater.cc:503] Master 127.0.98.62:41945 was elected leader, sending a full tablet report...
I20260929 19:01:09.705492 5496 ts_tablet_manager.cc:1428] T 6771d40d77af47b580939f3f15344912 P dc92236bb3cb43008381726c4f30b78f: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260929 19:01:09.705607 5219 heartbeater.cc:503] Master 127.0.98.62:41945 was elected leader, sending a full tablet report...
W20260929 19:01:09.712462 5220 tablet.cc:2369] T 6771d40d77af47b580939f3f15344912 P dc92236bb3cb43008381726c4f30b78f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 19:01:09.727645 5501 raft_consensus.cc:482] T 6771d40d77af47b580939f3f15344912 P 6d140f94dbbb429ea2f62e990effb1af [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 19:01:09.727762 5501 raft_consensus.cc:504] T 6771d40d77af47b580939f3f15344912 P 6d140f94dbbb429ea2f62e990effb1af [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d140f94dbbb429ea2f62e990effb1af" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33863 } } peers { permanent_uuid: "384ef3c1fb364c709a580fd51de6feec" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36699 } } peers { permanent_uuid: "dc92236bb3cb43008381726c4f30b78f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46775 } }
I20260929 19:01:09.728107 5501 leader_election.cc:288] T 6771d40d77af47b580939f3f15344912 P 6d140f94dbbb429ea2f62e990effb1af [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 384ef3c1fb364c709a580fd51de6feec (127.0.98.2:36699), dc92236bb3cb43008381726c4f30b78f (127.0.98.1:46775)
I20260929 19:01:09.729512 5483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:09.728266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46775 (local address 127.0.98.3:43677)
0929 19:01:09.728423 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:09.728435 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 19:01:09.728480 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:01:09.728647 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:01:09.728648 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:01:09.728650 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:01:09.729174 (+ 524us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 19:01:09.729175 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:01:09.729183 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 19:01:09.729198 (+ 15us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:01:09.729372 (+ 174us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:01:09.729373 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:01:09.729374 (+ 1us) client_negotiation.cc:769] Sending connection context
0929 19:01:09.729434 (+ 60us) client_negotiation.cc:240] Negotiation successful
0929 19:01:09.729448 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139}
I20260929 19:01:09.729547 5493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:09.728462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43677 (local address 127.0.98.1:46775)
0929 19:01:09.728524 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:09.728526 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:09.728535 (+ 9us) server_negotiation.cc:407] Connection header received
0929 19:01:09.728560 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:09.728561 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:09.728578 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:09.728632 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:01:09.729269 (+ 637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:01:09.729271 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:01:09.729271 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 19:01:09.729275 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:01:09.729293 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 19:01:09.729297 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:01:09.729357 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0929 19:01:09.729448 (+ 91us) server_negotiation.cc:299] Negotiation successful
0929 19:01:09.729463 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260929 19:01:09.729724 5494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:09.728377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38447 (local address 127.0.98.2:36699)
0929 19:01:09.728420 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:09.728422 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:09.728557 (+ 135us) server_negotiation.cc:407] Connection header received
0929 19:01:09.728647 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:09.728648 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:09.728653 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:09.728707 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:01:09.729467 (+ 760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:01:09.729468 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:01:09.729468 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 19:01:09.729472 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:01:09.729486 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 19:01:09.729490 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:01:09.729546 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0929 19:01:09.729639 (+ 93us) server_negotiation.cc:299] Negotiation successful
0929 19:01:09.729655 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260929 19:01:09.729842 5156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6771d40d77af47b580939f3f15344912" candidate_uuid: "6d140f94dbbb429ea2f62e990effb1af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc92236bb3cb43008381726c4f30b78f" is_pre_election: true
I20260929 19:01:09.729939 5504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:09.728266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36699 (local address 127.0.98.3:38447)
0929 19:01:09.728493 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:09.728504 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 19:01:09.728562 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:01:09.728728 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:01:09.728728 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:01:09.728730 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:01:09.729372 (+ 642us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 19:01:09.729372 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:01:09.729378 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 19:01:09.729389 (+ 11us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:01:09.729552 (+ 163us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:01:09.729553 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:01:09.729554 (+ 1us) client_negotiation.cc:769] Sending connection context
0929 19:01:09.729834 (+ 280us) client_negotiation.cc:240] Negotiation successful
0929 19:01:09.729857 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"mutex_wait_us":57,"thread_start_us":94,"threads_started":1}
I20260929 19:01:09.729977 5156 raft_consensus.cc:2454] T 6771d40d77af47b580939f3f15344912 P dc92236bb3cb43008381726c4f30b78f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d140f94dbbb429ea2f62e990effb1af in term 0.
I20260929 19:01:09.730202 5370 leader_election.cc:302] T 6771d40d77af47b580939f3f15344912 P 6d140f94dbbb429ea2f62e990effb1af [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d140f94dbbb429ea2f62e990effb1af, dc92236bb3cb43008381726c4f30b78f; no voters:
I20260929 19:01:09.730196 5304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6771d40d77af47b580939f3f15344912" candidate_uuid: "6d140f94dbbb429ea2f62e990effb1af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "384ef3c1fb364c709a580fd51de6feec" is_pre_election: true
I20260929 19:01:09.730325 5304 raft_consensus.cc:2454] T 6771d40d77af47b580939f3f15344912 P 384ef3c1fb364c709a580fd51de6feec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d140f94dbbb429ea2f62e990effb1af in term 0.
I20260929 19:01:09.730329 5501 raft_consensus.cc:2781] T 6771d40d77af47b580939f3f15344912 P 6d140f94dbbb429ea2f62e990effb1af [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 19:01:09.730399 5501 raft_consensus.cc:482] T 6771d40d77af47b580939f3f15344912 P 6d140f94dbbb429ea2f62e990effb1af [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 19:01:09.730432 5501 raft_consensus.cc:3037] T 6771d40d77af47b580939f3f15344912 P 6d140f94dbbb429ea2f62e990effb1af [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:01:09.731069 5501 raft_consensus.cc:504] T 6771d40d77af47b580939f3f15344912 P 6d140f94dbbb429ea2f62e990effb1af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d140f94dbbb429ea2f62e990effb1af" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33863 } } peers { permanent_uuid: "384ef3c1fb364c709a580fd51de6feec" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36699 } } peers { permanent_uuid: "dc92236bb3cb43008381726c4f30b78f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46775 } }
I20260929 19:01:09.731207 5501 leader_election.cc:288] T 6771d40d77af47b580939f3f15344912 P 6d140f94dbbb429ea2f62e990effb1af [CANDIDATE]: Term 1 election: Requested vote from peers 384ef3c1fb364c709a580fd51de6feec (127.0.98.2:36699), dc92236bb3cb43008381726c4f30b78f (127.0.98.1:46775)
I20260929 19:01:09.731319 5304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6771d40d77af47b580939f3f15344912" candidate_uuid: "6d140f94dbbb429ea2f62e990effb1af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "384ef3c1fb364c709a580fd51de6feec"
I20260929 19:01:09.731339 5156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6771d40d77af47b580939f3f15344912" candidate_uuid: "6d140f94dbbb429ea2f62e990effb1af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc92236bb3cb43008381726c4f30b78f"
I20260929 19:01:09.731403 5156 raft_consensus.cc:3037] T 6771d40d77af47b580939f3f15344912 P dc92236bb3cb43008381726c4f30b78f [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:01:09.731403 5304 raft_consensus.cc:3037] T 6771d40d77af47b580939f3f15344912 P 384ef3c1fb364c709a580fd51de6feec [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:01:09.732290 5304 raft_consensus.cc:2454] T 6771d40d77af47b580939f3f15344912 P 384ef3c1fb364c709a580fd51de6feec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d140f94dbbb429ea2f62e990effb1af in term 1.
I20260929 19:01:09.732290 5156 raft_consensus.cc:2454] T 6771d40d77af47b580939f3f15344912 P dc92236bb3cb43008381726c4f30b78f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d140f94dbbb429ea2f62e990effb1af in term 1.
I20260929 19:01:09.732451 5368 leader_election.cc:302] T 6771d40d77af47b580939f3f15344912 P 6d140f94dbbb429ea2f62e990effb1af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 384ef3c1fb364c709a580fd51de6feec, 6d140f94dbbb429ea2f62e990effb1af; no voters:
I20260929 19:01:09.732535 5501 raft_consensus.cc:2781] T 6771d40d77af47b580939f3f15344912 P 6d140f94dbbb429ea2f62e990effb1af [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:01:09.732668 5501 raft_consensus.cc:686] T 6771d40d77af47b580939f3f15344912 P 6d140f94dbbb429ea2f62e990effb1af [term 1 LEADER]: Becoming Leader. State: Replica: 6d140f94dbbb429ea2f62e990effb1af, State: Running, Role: LEADER
I20260929 19:01:09.732889 5501 consensus_queue.cc:237] T 6771d40d77af47b580939f3f15344912 P 6d140f94dbbb429ea2f62e990effb1af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d140f94dbbb429ea2f62e990effb1af" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33863 } } peers { permanent_uuid: "384ef3c1fb364c709a580fd51de6feec" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36699 } } peers { permanent_uuid: "dc92236bb3cb43008381726c4f30b78f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46775 } }
I20260929 19:01:09.733709 5030 catalog_manager.cc:5495] T 6771d40d77af47b580939f3f15344912 P 6d140f94dbbb429ea2f62e990effb1af reported cstate change: term changed from 0 to 1, leader changed from <none> to 6d140f94dbbb429ea2f62e990effb1af (127.0.98.3). New cstate: current_term: 1 leader_uuid: "6d140f94dbbb429ea2f62e990effb1af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d140f94dbbb429ea2f62e990effb1af" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "384ef3c1fb364c709a580fd51de6feec" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36699 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dc92236bb3cb43008381726c4f30b78f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46775 } health_report { overall_health: UNKNOWN } } }
I20260929 19:01:09.741410 5491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:09.740518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43514 (local address 127.0.98.3:33863)
0929 19:01:09.740584 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:09.740585 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:09.740592 (+ 7us) server_negotiation.cc:407] Connection header received
0929 19:01:09.740631 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:09.740632 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:09.740640 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:09.740687 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:01:09.741231 (+ 544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:01:09.741232 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:01:09.741232 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 19:01:09.741235 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:01:09.741245 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 19:01:09.741248 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:01:09.741288 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0929 19:01:09.741316 (+ 28us) server_negotiation.cc:299] Negotiation successful
0929 19:01:09.741326 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 19:01:09.742831 5156 raft_consensus.cc:1261] T 6771d40d77af47b580939f3f15344912 P dc92236bb3cb43008381726c4f30b78f [term 1 FOLLOWER]: Refusing update from remote peer 6d140f94dbbb429ea2f62e990effb1af: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 19:01:09.742887 5304 raft_consensus.cc:1261] T 6771d40d77af47b580939f3f15344912 P 384ef3c1fb364c709a580fd51de6feec [term 1 FOLLOWER]: Refusing update from remote peer 6d140f94dbbb429ea2f62e990effb1af: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 19:01:09.743139 5505 consensus_queue.cc:1035] T 6771d40d77af47b580939f3f15344912 P 6d140f94dbbb429ea2f62e990effb1af [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc92236bb3cb43008381726c4f30b78f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 19:01:09.743238 5501 consensus_queue.cc:1035] T 6771d40d77af47b580939f3f15344912 P 6d140f94dbbb429ea2f62e990effb1af [LEADER]: Connected to new peer: Peer: permanent_uuid: "384ef3c1fb364c709a580fd51de6feec" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36699 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 19:01:09.746039 5508 mvcc.cc:205] Tried to move back new op lower bound from 7334741892064620544 to 7334741892026773504. Current Snapshot: MvccSnapshot[applied={T|T < 7334741892064620544}]
I20260929 19:01:09.746166 5509 mvcc.cc:205] Tried to move back new op lower bound from 7334741892064620544 to 7334741892026773504. Current Snapshot: MvccSnapshot[applied={T|T < 7334741892064620544}]
I20260929 19:01:09.746706 5510 mvcc.cc:205] Tried to move back new op lower bound from 7334741892064620544 to 7334741892026773504. Current Snapshot: MvccSnapshot[applied={T|T < 7334741892064620544}]
I20260929 19:01:09.748232 5030 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43540:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 19:01:09.748327 5030 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43540:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 19:01:09.748981 5030 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 16e8738dab0e402ca5f53d9b41c1c57c: Sending DeleteTablet for 3 replicas of tablet 6771d40d77af47b580939f3f15344912
I20260929 19:01:09.749274 5415 tablet_service.cc:1514] Processing DeleteTablet for tablet 6771d40d77af47b580939f3f15344912 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:01:09 UTC) from {username='slave'} at 127.0.0.1:43506
I20260929 19:01:09.749320 5284 tablet_service.cc:1514] Processing DeleteTablet for tablet 6771d40d77af47b580939f3f15344912 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:01:09 UTC) from {username='slave'} at 127.0.0.1:49470
I20260929 19:01:09.749388 5144 tablet_service.cc:1514] Processing DeleteTablet for tablet 6771d40d77af47b580939f3f15344912 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:01:09 UTC) from {username='slave'} at 127.0.0.1:56480
I20260929 19:01:09.749543 5518 tablet_replica.cc:330] T 6771d40d77af47b580939f3f15344912 P 384ef3c1fb364c709a580fd51de6feec: stopping tablet replica
I20260929 19:01:09.749560 5519 tablet_replica.cc:330] T 6771d40d77af47b580939f3f15344912 P dc92236bb3cb43008381726c4f30b78f: stopping tablet replica
I20260929 19:01:09.749627 5518 raft_consensus.cc:2229] T 6771d40d77af47b580939f3f15344912 P 384ef3c1fb364c709a580fd51de6feec [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 19:01:09.749641 5519 raft_consensus.cc:2229] T 6771d40d77af47b580939f3f15344912 P dc92236bb3cb43008381726c4f30b78f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 19:01:09.749714 5518 raft_consensus.cc:2258] T 6771d40d77af47b580939f3f15344912 P 384ef3c1fb364c709a580fd51de6feec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 19:01:09.749732 5519 raft_consensus.cc:2258] T 6771d40d77af47b580939f3f15344912 P dc92236bb3cb43008381726c4f30b78f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 19:01:09.749802 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUbnLHE/build/release/bin/kudu with pid 5091
I20260929 19:01:09.750510 5518 ts_tablet_manager.cc:1905] T 6771d40d77af47b580939f3f15344912 P 384ef3c1fb364c709a580fd51de6feec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 19:01:09.752008 5518 ts_tablet_manager.cc:1918] T 6771d40d77af47b580939f3f15344912 P 384ef3c1fb364c709a580fd51de6feec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 19:01:09.752076 5518 log.cc:1196] T 6771d40d77af47b580939f3f15344912 P 384ef3c1fb364c709a580fd51de6feec: Deleting WAL directory at /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708447806542-392-0/minicluster-data/ts-1/wal/wals/6771d40d77af47b580939f3f15344912
I20260929 19:01:09.752326 5518 ts_tablet_manager.cc:1939] T 6771d40d77af47b580939f3f15344912 P 384ef3c1fb364c709a580fd51de6feec: Deleting consensus metadata
I20260929 19:01:09.752732 5015 catalog_manager.cc:4841] TS 384ef3c1fb364c709a580fd51de6feec (127.0.98.2:36699): tablet 6771d40d77af47b580939f3f15344912 (table test-table [id=306d5938d84d411ba63f079e55d7121b]) successfully deleted
W20260929 19:01:09.755179 5017 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:46775 (error 108)
I20260929 19:01:09.755488 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUbnLHE/build/release/bin/kudu with pid 5222
I20260929 19:01:09.755499 5517 tablet_replica.cc:330] T 6771d40d77af47b580939f3f15344912 P 6d140f94dbbb429ea2f62e990effb1af: stopping tablet replica
I20260929 19:01:09.756270 5517 raft_consensus.cc:2229] T 6771d40d77af47b580939f3f15344912 P 6d140f94dbbb429ea2f62e990effb1af [term 1 LEADER]: Raft consensus shutting down.
I20260929 19:01:09.756402 5517 raft_consensus.cc:2258] T 6771d40d77af47b580939f3f15344912 P 6d140f94dbbb429ea2f62e990effb1af [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 19:01:09.756685 5490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:09.756577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46775 (local address 127.0.0.1:56492)
0929 19:01:09.756616 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:09.756657 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
I20260929 19:01:09.756850 5517 ts_tablet_manager.cc:1905] T 6771d40d77af47b580939f3f15344912 P 6d140f94dbbb429ea2f62e990effb1af: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260929 19:01:09.756851 5017 catalog_manager.cc:4584] TS dc92236bb3cb43008381726c4f30b78f (127.0.98.1:46775): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6771d40d77af47b580939f3f15344912: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46775: connect: Connection refused (error 111)
I20260929 19:01:09.758872 5517 ts_tablet_manager.cc:1918] T 6771d40d77af47b580939f3f15344912 P 6d140f94dbbb429ea2f62e990effb1af: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 19:01:09.758934 5517 log.cc:1196] T 6771d40d77af47b580939f3f15344912 P 6d140f94dbbb429ea2f62e990effb1af: Deleting WAL directory at /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790708447806542-392-0/minicluster-data/ts-2/wal/wals/6771d40d77af47b580939f3f15344912
I20260929 19:01:09.759198 5517 ts_tablet_manager.cc:1939] T 6771d40d77af47b580939f3f15344912 P 6d140f94dbbb429ea2f62e990effb1af: Deleting consensus metadata
I20260929 19:01:09.759703 5015 catalog_manager.cc:4841] TS 6d140f94dbbb429ea2f62e990effb1af (127.0.98.3:33863): tablet 6771d40d77af47b580939f3f15344912 (table test-table [id=306d5938d84d411ba63f079e55d7121b]) successfully deleted
I20260929 19:01:09.761140 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUbnLHE/build/release/bin/kudu with pid 5353
I20260929 19:01:09.764982 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUbnLHE/build/release/bin/kudu with pid 5000
2026-09-29T19:01:09Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (571 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260929 19:01:09.784678 392 mini_oidc.cc:152] Starting JWKS server
I20260929 19:01:09.785729 392 webserver.cc:460] Webserver started at https://127.0.0.1:37443/ using document root <none> and password file <none>
I20260929 19:01:09.785800 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260929 19:01:09.785919 392 webserver.cc:460] Webserver started at http://127.0.0.1:42985/ using document root <none> and password file <none>
2026-09-29T19:01:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T19:01:09Z Disabled control of system clock
I20260929 19:01:09.793099 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUbnLHE/build/release/bin/kudu
/tmp/dist-test-taskUbnLHE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790708447806542-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790708447806542-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790708447806542-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790708447806542-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40679
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:33225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40679
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37443/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790708447806542-392-0/testchainca.cert with env {}
W20260929 19:01:09.869874 5526 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:01:09.870057 5526 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:01:09.870078 5526 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:01:09.871122 5526 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 19:01:09.871153 5526 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:01:09.871165 5526 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:01:09.871178 5526 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 19:01:09.871191 5526 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 19:01:09.871202 5526 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 19:01:09.871222 5526 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:37443/jwks.json
I20260929 19:01:09.872635 5526 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790708447806542-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790708447806542-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40679
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40679
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37443/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790708447806542-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790708447806542-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5526
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790708447806542-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 18:09:13 UTC on ccycloud.quasar-hpkalf.root.comops.site
build id 4680913
I20260929 19:01:09.872851 5526 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:01:09.873027 5526 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:01:09.877435 5534 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:01:09.877424 5532 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:01:09.877425 5531 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:01:09.877647 5526 server_base.cc:1034] running on GCE node
I20260929 19:01:09.877890 5526 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:01:09.878211 5526 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:01:09.879366 5526 hybrid_clock.cc:648] HybridClock initialized: now 1790708469879361 us; error 35 us; skew 500 ppm
I20260929 19:01:09.880499 5526 webserver.cc:460] Webserver started at http://127.0.98.62:37287/ using document root <none> and password file <none>
I20260929 19:01:09.880724 5526 fs_manager.cc:359] Metadata directory not provided
I20260929 19:01:09.880795 5526 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:01:09.880952 5526 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:01:09.881918 5526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790708447806542-392-0/minicluster-data/master-0/data/instance:
uuid: "ff5601ac4c7e46a9b91c512c5deff5e7"
format_stamp: "Formatted at 2026-09-29 19:01:09 on dist-test-slave-lqkv"
server_key: "5642183ff705d2a692b310a93078954f"
server_key_iv: "8b388843b7bc5e121dcbd773d0f5c335"
server_key_version: "encryptionkey@0"
I20260929 19:01:09.882261 5526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790708447806542-392-0/minicluster-data/master-0/wal/instance:
uuid: "ff5601ac4c7e46a9b91c512c5deff5e7"
format_stamp: "Formatted at 2026-09-29 19:01:09 on dist-test-slave-lqkv"
server_key: "5642183ff705d2a692b310a93078954f"
server_key_iv: "8b388843b7bc5e121dcbd773d0f5c335"
server_key_version: "encryptionkey@0"
I20260929 19:01:09.883472 5526 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 19:01:09.884100 5540 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:09.884258 5526 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 19:01:09.884346 5526 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790708447806542-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790708447806542-392-0/minicluster-data/master-0/wal
uuid: "ff5601ac4c7e46a9b91c512c5deff5e7"
format_stamp: "Formatted at 2026-09-29 19:01:09 on dist-test-slave-lqkv"
server_key: "5642183ff705d2a692b310a93078954f"
server_key_iv: "8b388843b7bc5e121dcbd773d0f5c335"
server_key_version: "encryptionkey@0"
I20260929 19:01:09.884428 5526 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790708447806542-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790708447806542-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790708447806542-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:01:09.895660 5526 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:01:09.895926 5526 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:01:09.896065 5526 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:01:09.899689 5526 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40679
I20260929 19:01:09.899719 5592 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40679 every 8 connection(s)
I20260929 19:01:09.900019 5526 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790708447806542-392-0/minicluster-data/master-0/data/info.pb
I20260929 19:01:09.900910 5593 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:01:09.902933 5593 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff5601ac4c7e46a9b91c512c5deff5e7: Bootstrap starting.
I20260929 19:01:09.903481 5593 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ff5601ac4c7e46a9b91c512c5deff5e7: Neither blocks nor log segments found. Creating new log.
I20260929 19:01:09.903764 5593 log.cc:824] T 00000000000000000000000000000000 P ff5601ac4c7e46a9b91c512c5deff5e7: Log is configured to *not* fsync() on all Append() calls
I20260929 19:01:09.904371 5593 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff5601ac4c7e46a9b91c512c5deff5e7: No bootstrap required, opened a new log
I20260929 19:01:09.905920 5593 raft_consensus.cc:348] T 00000000000000000000000000000000 P ff5601ac4c7e46a9b91c512c5deff5e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff5601ac4c7e46a9b91c512c5deff5e7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40679 } }
I20260929 19:01:09.906042 5593 raft_consensus.cc:374] T 00000000000000000000000000000000 P ff5601ac4c7e46a9b91c512c5deff5e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:01:09.906064 5593 raft_consensus.cc:729] T 00000000000000000000000000000000 P ff5601ac4c7e46a9b91c512c5deff5e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff5601ac4c7e46a9b91c512c5deff5e7, State: Initialized, Role: FOLLOWER
I20260929 19:01:09.906229 5593 consensus_queue.cc:260] T 00000000000000000000000000000000 P ff5601ac4c7e46a9b91c512c5deff5e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff5601ac4c7e46a9b91c512c5deff5e7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40679 } }
I20260929 19:01:09.906323 5593 raft_consensus.cc:388] T 00000000000000000000000000000000 P ff5601ac4c7e46a9b91c512c5deff5e7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 19:01:09.906369 5593 raft_consensus.cc:482] T 00000000000000000000000000000000 P ff5601ac4c7e46a9b91c512c5deff5e7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 19:01:09.906417 5593 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ff5601ac4c7e46a9b91c512c5deff5e7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:01:09.906665 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUbnLHE/build/release/bin/kudu as pid 5526
I20260929 19:01:09.906754 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790708447806542-392-0/minicluster-data/master-0/wal/instance
I20260929 19:01:09.906934 392 external_mini_cluster.cc:1428] Setting key 7c683215dd2ff88cb8993a831a52bf65
I20260929 19:01:09.906980 5593 raft_consensus.cc:504] T 00000000000000000000000000000000 P ff5601ac4c7e46a9b91c512c5deff5e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff5601ac4c7e46a9b91c512c5deff5e7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40679 } }
I20260929 19:01:09.907079 5593 leader_election.cc:302] T 00000000000000000000000000000000 P ff5601ac4c7e46a9b91c512c5deff5e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ff5601ac4c7e46a9b91c512c5deff5e7; no voters:
I20260929 19:01:09.907251 5593 leader_election.cc:288] T 00000000000000000000000000000000 P ff5601ac4c7e46a9b91c512c5deff5e7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 19:01:09.907449 5598 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ff5601ac4c7e46a9b91c512c5deff5e7 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:01:09.907502 5593 sys_catalog.cc:564] T 00000000000000000000000000000000 P ff5601ac4c7e46a9b91c512c5deff5e7 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 19:01:09.907618 5598 raft_consensus.cc:686] T 00000000000000000000000000000000 P ff5601ac4c7e46a9b91c512c5deff5e7 [term 1 LEADER]: Becoming Leader. State: Replica: ff5601ac4c7e46a9b91c512c5deff5e7, State: Running, Role: LEADER
I20260929 19:01:09.907704 5598 consensus_queue.cc:237] T 00000000000000000000000000000000 P ff5601ac4c7e46a9b91c512c5deff5e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff5601ac4c7e46a9b91c512c5deff5e7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40679 } }
I20260929 19:01:09.908160 5596 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff5601ac4c7e46a9b91c512c5deff5e7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ff5601ac4c7e46a9b91c512c5deff5e7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff5601ac4c7e46a9b91c512c5deff5e7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40679 } } }
I20260929 19:01:09.908253 5596 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff5601ac4c7e46a9b91c512c5deff5e7 [sys.catalog]: This master's current role is: LEADER
I20260929 19:01:09.908578 5600 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff5601ac4c7e46a9b91c512c5deff5e7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ff5601ac4c7e46a9b91c512c5deff5e7. Latest consensus state: current_term: 1 leader_uuid: "ff5601ac4c7e46a9b91c512c5deff5e7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff5601ac4c7e46a9b91c512c5deff5e7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40679 } } }
I20260929 19:01:09.908659 5600 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff5601ac4c7e46a9b91c512c5deff5e7 [sys.catalog]: This master's current role is: LEADER
I20260929 19:01:09.908581 5607 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 19:01:09.909265 5607 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 19:01:09.910902 5607 catalog_manager.cc:1347] Generated new cluster ID: 58c620c0bf1b41799a99d172deb0c149
I20260929 19:01:09.910954 5607 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 19:01:09.913635 5599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:09.907326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56120 (local address 127.0.98.62:40679)
0929 19:01:09.908947 (+ 1621us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:09.908955 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:09.908988 (+ 33us) server_negotiation.cc:407] Connection header received
0929 19:01:09.909094 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:09.909100 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:09.909159 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:09.909330 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:01:09.909571 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:09.910937 (+ 1366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:09.911797 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:09.912225 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:09.912348 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:09.912971 (+ 623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:01:09.912974 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:01:09.912974 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 19:01:09.913141 (+ 167us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:01:09.913181 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 19:01:09.913191 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:01:09.913260 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0929 19:01:09.913357 (+ 97us) server_negotiation.cc:299] Negotiation successful
0929 19:01:09.913441 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1541,"thread_start_us":66,"threads_started":1}
I20260929 19:01:09.914358 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 19:01:09.945024 5607 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 19:01:09.945721 5607 catalog_manager.cc:1504] Loading token signing keys...
I20260929 19:01:09.952330 5607 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ff5601ac4c7e46a9b91c512c5deff5e7: Generated new TSK 0
I20260929 19:01:09.952491 5607 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260929 19:01:09.976771 5623 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260929 19:01:09.976923 5623 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260929 19:01:09.977166 5599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:09.962244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56126 (local address 127.0.98.62:40679)
0929 19:01:09.962286 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:09.962288 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:09.962405 (+ 117us) server_negotiation.cc:407] Connection header received
0929 19:01:09.962466 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:09.962467 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:09.962478 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:09.962564 (+ 86us) server_negotiation.cc:226] Negotiated authn=JWT
0929 19:01:09.962676 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:09.963399 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:09.964111 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:09.964399 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:09.964463 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:09.964628 (+ 165us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 19:01:09.976963 (+ 12335us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37443/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0929 19:01:09.977093 (+ 130us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56126: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37443/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":24}
W20260929 19:01:09.977221 5599 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56126: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37443/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260929 19:01:09.977494 5622 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:01:09.962167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40679 (local address 127.0.0.1:56126)
0929 19:01:09.962335 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:09.962344 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 19:01:09.962395 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:01:09.962563 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:01:09.962564 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:01:09.962567 (+ 3us) client_negotiation.cc:189] Negotiated authn=JWT
0929 19:01:09.962617 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:09.962618 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:09.963487 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:09.963487 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:09.964030 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:09.964032 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:09.964474 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:09.964477 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:09.964518 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:09.964532 (+ 14us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 19:01:09.977091 (+ 12559us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37443/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0929 19:01:09.977185 (+ 94us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:40679: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37443/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":93,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.98.62:40679: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37443/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260929 19:01:09.977917 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUbnLHE/build/release/bin/kudu with pid 5526
2026-09-29T19:01:09Z chronyd exiting
I20260929 19:01:10.013792 392 test_util.cc:175] -----------------------------------------------
I20260929 19:01:10.013855 392 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790708447806542-392-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (230 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260929 19:01:10.014497 392 mini_oidc.cc:152] Starting JWKS server
I20260929 19:01:10.015082 392 webserver.cc:460] Webserver started at https://127.0.0.1:33051/ using document root <none> and password file <none>
I20260929 19:01:10.015120 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260929 19:01:10.015249 392 webserver.cc:460] Webserver started at http://127.0.0.1:46745/ using document root <none> and password file <none>
2026-09-29T19:01:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T19:01:10Z Disabled control of system clock
I20260929 19:01:10.022068 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUbnLHE/build/release/bin/kudu
/tmp/dist-test-taskUbnLHE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790708447806542-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790708447806542-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790708447806542-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790708447806542-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33609
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:39239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33609
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33051/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790708447806542-392-0/testchainca.cert with env {}
W20260929 19:01:10.098654 5629 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:01:10.098888 5629 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:01:10.098924 5629 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:01:10.100061 5629 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 19:01:10.100124 5629 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:01:10.100148 5629 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:01:10.100179 5629 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 19:01:10.100203 5629 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 19:01:10.100234 5629 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 19:01:10.100261 5629 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:33051/jwks.json
I20260929 19:01:10.101722 5629 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790708447806542-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790708447806542-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33609
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33609
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33051/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790708447806542-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790708447806542-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5629
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790708447806542-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 18:09:13 UTC on ccycloud.quasar-hpkalf.root.comops.site
build id 4680913
I20260929 19:01:10.102048 5629 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:01:10.102252 5629 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 19:01:10.106989 5629 server_base.cc:1034] running on GCE node
W20260929 19:01:10.106912 5637 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:01:10.106848 5634 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:01:10.106865 5635 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:01:10.107393 5629 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:01:10.107663 5629 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:01:10.108879 5629 hybrid_clock.cc:648] HybridClock initialized: now 1790708470108859 us; error 40 us; skew 500 ppm
I20260929 19:01:10.110074 5629 webserver.cc:460] Webserver started at http://127.0.98.62:38703/ using document root <none> and password file <none>
I20260929 19:01:10.110335 5629 fs_manager.cc:359] Metadata directory not provided
I20260929 19:01:10.110397 5629 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:01:10.110525 5629 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:01:10.111490 5629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790708447806542-392-0/minicluster-data/master-0/data/instance:
uuid: "6a475ad529274217b0787e9e721a9075"
format_stamp: "Formatted at 2026-09-29 19:01:10 on dist-test-slave-lqkv"
server_key: "3caf80c008a968eb6fe5b726b32c88d5"
server_key_iv: "c71353ea03132f00bc74e2a4e1f1f3cd"
server_key_version: "encryptionkey@0"
I20260929 19:01:10.111869 5629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790708447806542-392-0/minicluster-data/master-0/wal/instance:
uuid: "6a475ad529274217b0787e9e721a9075"
format_stamp: "Formatted at 2026-09-29 19:01:10 on dist-test-slave-lqkv"
server_key: "3caf80c008a968eb6fe5b726b32c88d5"
server_key_iv: "c71353ea03132f00bc74e2a4e1f1f3cd"
server_key_version: "encryptionkey@0"
I20260929 19:01:10.113087 5629 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 19:01:10.113740 5643 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:10.113906 5629 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 19:01:10.113996 5629 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790708447806542-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790708447806542-392-0/minicluster-data/master-0/wal
uuid: "6a475ad529274217b0787e9e721a9075"
format_stamp: "Formatted at 2026-09-29 19:01:10 on dist-test-slave-lqkv"
server_key: "3caf80c008a968eb6fe5b726b32c88d5"
server_key_iv: "c71353ea03132f00bc74e2a4e1f1f3cd"
server_key_version: "encryptionkey@0"
I20260929 19:01:10.114068 5629 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790708447806542-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790708447806542-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790708447806542-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:01:10.137022 5629 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:01:10.137310 5629 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:01:10.137466 5629 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:01:10.141327 5629 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33609
I20260929 19:01:10.141360 5695 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33609 every 8 connection(s)
I20260929 19:01:10.141705 5629 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790708447806542-392-0/minicluster-data/master-0/data/info.pb
I20260929 19:01:10.142693 5696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:01:10.144862 5696 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a475ad529274217b0787e9e721a9075: Bootstrap starting.
I20260929 19:01:10.145403 5696 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6a475ad529274217b0787e9e721a9075: Neither blocks nor log segments found. Creating new log.
I20260929 19:01:10.145677 5696 log.cc:824] T 00000000000000000000000000000000 P 6a475ad529274217b0787e9e721a9075: Log is configured to *not* fsync() on all Append() calls
I20260929 19:01:10.145915 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUbnLHE/build/release/bin/kudu as pid 5629
I20260929 19:01:10.146034 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790708447806542-392-0/minicluster-data/master-0/wal/instance
I20260929 19:01:10.146229 392 external_mini_cluster.cc:1428] Setting key 1685aaea228342c145cf9d0c9906a2ff
I20260929 19:01:10.146482 5696 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a475ad529274217b0787e9e721a9075: No bootstrap required, opened a new log
I20260929 19:01:10.148070 5696 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6a475ad529274217b0787e9e721a9075 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a475ad529274217b0787e9e721a9075" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33609 } }
I20260929 19:01:10.148212 5696 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6a475ad529274217b0787e9e721a9075 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:01:10.148268 5696 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6a475ad529274217b0787e9e721a9075 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a475ad529274217b0787e9e721a9075, State: Initialized, Role: FOLLOWER
I20260929 19:01:10.148422 5696 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6a475ad529274217b0787e9e721a9075 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a475ad529274217b0787e9e721a9075" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33609 } }
I20260929 19:01:10.148505 5696 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6a475ad529274217b0787e9e721a9075 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 19:01:10.148537 5696 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6a475ad529274217b0787e9e721a9075 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 19:01:10.148586 5696 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6a475ad529274217b0787e9e721a9075 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:01:10.149266 5696 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6a475ad529274217b0787e9e721a9075 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a475ad529274217b0787e9e721a9075" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33609 } }
I20260929 19:01:10.149374 5696 leader_election.cc:302] T 00000000000000000000000000000000 P 6a475ad529274217b0787e9e721a9075 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6a475ad529274217b0787e9e721a9075; no voters:
I20260929 19:01:10.149505 5696 leader_election.cc:288] T 00000000000000000000000000000000 P 6a475ad529274217b0787e9e721a9075 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 19:01:10.149577 5701 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6a475ad529274217b0787e9e721a9075 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:01:10.149732 5701 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6a475ad529274217b0787e9e721a9075 [term 1 LEADER]: Becoming Leader. State: Replica: 6a475ad529274217b0787e9e721a9075, State: Running, Role: LEADER
I20260929 19:01:10.149740 5696 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6a475ad529274217b0787e9e721a9075 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 19:01:10.149824 5701 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6a475ad529274217b0787e9e721a9075 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a475ad529274217b0787e9e721a9075" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33609 } }
I20260929 19:01:10.150132 5702 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a475ad529274217b0787e9e721a9075 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6a475ad529274217b0787e9e721a9075" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a475ad529274217b0787e9e721a9075" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33609 } } }
I20260929 19:01:10.150121 5701 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a475ad529274217b0787e9e721a9075 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6a475ad529274217b0787e9e721a9075. Latest consensus state: current_term: 1 leader_uuid: "6a475ad529274217b0787e9e721a9075" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a475ad529274217b0787e9e721a9075" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33609 } } }
I20260929 19:01:10.150250 5702 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a475ad529274217b0787e9e721a9075 [sys.catalog]: This master's current role is: LEADER
I20260929 19:01:10.150257 5701 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a475ad529274217b0787e9e721a9075 [sys.catalog]: This master's current role is: LEADER
I20260929 19:01:10.151052 5708 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 19:01:10.151500 5708 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 19:01:10.153098 5708 catalog_manager.cc:1347] Generated new cluster ID: 8e5259031e614bfd980133e3aafb28f1
I20260929 19:01:10.153146 5708 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 19:01:10.153519 5700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:10.146654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42262 (local address 127.0.98.62:33609)
0929 19:01:10.146861 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:10.146868 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:10.146903 (+ 35us) server_negotiation.cc:407] Connection header received
0929 19:01:10.148056 (+ 1153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:10.148061 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:10.148109 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:10.148227 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:01:10.148409 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:10.150061 (+ 1652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:10.151324 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:10.151901 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:10.152018 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:10.152710 (+ 692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:01:10.152713 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:01:10.152713 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 19:01:10.152932 (+ 219us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:01:10.152963 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 19:01:10.152973 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:01:10.153045 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 19:01:10.153259 (+ 214us) server_negotiation.cc:299] Negotiation successful
0929 19:01:10.153340 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":79,"threads_started":1}
I20260929 19:01:10.154383 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 19:01:10.171989 5708 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 19:01:10.172683 5708 catalog_manager.cc:1504] Loading token signing keys...
I20260929 19:01:10.178701 5708 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6a475ad529274217b0787e9e721a9075: Generated new TSK 0
I20260929 19:01:10.178867 5708 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260929 19:01:10.202399 5726 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260929 19:01:10.202461 5726 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260929 19:01:10.202735 5700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:10.189226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42266 (local address 127.0.98.62:33609)
0929 19:01:10.189279 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:10.189281 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:10.189350 (+ 69us) server_negotiation.cc:407] Connection header received
0929 19:01:10.189405 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:10.189406 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:10.189417 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:10.189487 (+ 70us) server_negotiation.cc:226] Negotiated authn=JWT
0929 19:01:10.189602 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:10.190334 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:10.191032 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:10.191323 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:10.191387 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:10.191525 (+ 138us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 19:01:10.202547 (+ 11022us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33051/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0929 19:01:10.202668 (+ 121us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:42266: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33051/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":31}
W20260929 19:01:10.202785 5700 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:42266: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33051/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260929 19:01:10.202822 5725 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:01:10.189150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33609 (local address 127.0.0.1:42266)
0929 19:01:10.189295 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:10.189305 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 19:01:10.189357 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:01:10.189497 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:01:10.189497 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:01:10.189499 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0929 19:01:10.189543 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:10.189544 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:10.190405 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:10.190406 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:10.190961 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:10.190963 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:10.191393 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:10.191395 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:10.191434 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:10.191438 (+ 4us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 19:01:10.202655 (+ 11217us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33051/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0929 19:01:10.202728 (+ 73us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:33609: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33051/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":93,"threads_started":1}
I20260929 19:01:10.203254 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUbnLHE/build/release/bin/kudu with pid 5629
2026-09-29T19:01:10Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (225 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260929 19:01:10.240633 392 mini_oidc.cc:152] Starting JWKS server
I20260929 19:01:10.241353 392 webserver.cc:460] Webserver started at https://127.0.0.1:40707/ using document root <none> and password file <none>
I20260929 19:01:10.241418 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260929 19:01:10.241556 392 webserver.cc:460] Webserver started at http://127.0.0.1:34403/ using document root <none> and password file <none>
2026-09-29T19:01:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T19:01:10Z Disabled control of system clock
I20260929 19:01:10.249049 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUbnLHE/build/release/bin/kudu
/tmp/dist-test-taskUbnLHE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790708447806542-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790708447806542-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790708447806542-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790708447806542-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41935
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:37831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41935
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40707/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260929 19:01:10.334240 5732 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:01:10.334443 5732 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:01:10.334465 5732 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:01:10.335527 5732 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 19:01:10.335561 5732 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:01:10.335574 5732 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:01:10.335587 5732 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 19:01:10.335599 5732 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 19:01:10.335611 5732 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 19:01:10.335623 5732 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:40707/jwks.json
I20260929 19:01:10.336934 5732 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790708447806542-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790708447806542-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41935
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41935
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40707/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790708447806542-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5732
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790708447806542-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 18:09:13 UTC on ccycloud.quasar-hpkalf.root.comops.site
build id 4680913
I20260929 19:01:10.337105 5732 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:01:10.337337 5732 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:01:10.341944 5738 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:01:10.341952 5737 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:01:10.341959 5732 server_base.cc:1034] running on GCE node
W20260929 19:01:10.342061 5740 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:01:10.342475 5732 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:01:10.342798 5732 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:01:10.343958 5732 hybrid_clock.cc:648] HybridClock initialized: now 1790708470343955 us; error 53 us; skew 500 ppm
I20260929 19:01:10.345227 5732 webserver.cc:460] Webserver started at http://127.0.98.62:35097/ using document root <none> and password file <none>
I20260929 19:01:10.345463 5732 fs_manager.cc:359] Metadata directory not provided
I20260929 19:01:10.345530 5732 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:01:10.345664 5732 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:01:10.346930 5732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790708447806542-392-0/minicluster-data/master-0/data/instance:
uuid: "9d6976ca78214d76a23cdd0b09863f60"
format_stamp: "Formatted at 2026-09-29 19:01:10 on dist-test-slave-lqkv"
server_key: "7e159f1b919187fe53100521a06afa23"
server_key_iv: "a0493c2cd9240518935a8c31169b8d28"
server_key_version: "encryptionkey@0"
I20260929 19:01:10.347379 5732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790708447806542-392-0/minicluster-data/master-0/wal/instance:
uuid: "9d6976ca78214d76a23cdd0b09863f60"
format_stamp: "Formatted at 2026-09-29 19:01:10 on dist-test-slave-lqkv"
server_key: "7e159f1b919187fe53100521a06afa23"
server_key_iv: "a0493c2cd9240518935a8c31169b8d28"
server_key_version: "encryptionkey@0"
I20260929 19:01:10.349072 5732 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260929 19:01:10.349973 5746 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:10.350176 5732 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 19:01:10.350270 5732 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790708447806542-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790708447806542-392-0/minicluster-data/master-0/wal
uuid: "9d6976ca78214d76a23cdd0b09863f60"
format_stamp: "Formatted at 2026-09-29 19:01:10 on dist-test-slave-lqkv"
server_key: "7e159f1b919187fe53100521a06afa23"
server_key_iv: "a0493c2cd9240518935a8c31169b8d28"
server_key_version: "encryptionkey@0"
I20260929 19:01:10.350354 5732 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790708447806542-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790708447806542-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790708447806542-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:01:10.358057 5732 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:01:10.358327 5732 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:01:10.358464 5732 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:01:10.362272 5732 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41935
I20260929 19:01:10.362308 5798 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41935 every 8 connection(s)
I20260929 19:01:10.362706 5732 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790708447806542-392-0/minicluster-data/master-0/data/info.pb
I20260929 19:01:10.363368 5799 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:01:10.366031 5799 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d6976ca78214d76a23cdd0b09863f60: Bootstrap starting.
I20260929 19:01:10.366590 5799 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9d6976ca78214d76a23cdd0b09863f60: Neither blocks nor log segments found. Creating new log.
I20260929 19:01:10.366869 5799 log.cc:824] T 00000000000000000000000000000000 P 9d6976ca78214d76a23cdd0b09863f60: Log is configured to *not* fsync() on all Append() calls
I20260929 19:01:10.367496 5799 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d6976ca78214d76a23cdd0b09863f60: No bootstrap required, opened a new log
I20260929 19:01:10.368992 5799 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9d6976ca78214d76a23cdd0b09863f60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d6976ca78214d76a23cdd0b09863f60" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41935 } }
I20260929 19:01:10.369128 5799 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9d6976ca78214d76a23cdd0b09863f60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:01:10.369169 5799 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9d6976ca78214d76a23cdd0b09863f60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d6976ca78214d76a23cdd0b09863f60, State: Initialized, Role: FOLLOWER
I20260929 19:01:10.369297 5799 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9d6976ca78214d76a23cdd0b09863f60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d6976ca78214d76a23cdd0b09863f60" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41935 } }
I20260929 19:01:10.369369 5799 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9d6976ca78214d76a23cdd0b09863f60 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 19:01:10.369410 5799 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9d6976ca78214d76a23cdd0b09863f60 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 19:01:10.369457 5799 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9d6976ca78214d76a23cdd0b09863f60 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:01:10.369946 5799 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9d6976ca78214d76a23cdd0b09863f60 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d6976ca78214d76a23cdd0b09863f60" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41935 } }
I20260929 19:01:10.370051 5799 leader_election.cc:302] T 00000000000000000000000000000000 P 9d6976ca78214d76a23cdd0b09863f60 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9d6976ca78214d76a23cdd0b09863f60; no voters:
I20260929 19:01:10.370216 5799 leader_election.cc:288] T 00000000000000000000000000000000 P 9d6976ca78214d76a23cdd0b09863f60 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 19:01:10.370267 5802 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9d6976ca78214d76a23cdd0b09863f60 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:01:10.370465 5802 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9d6976ca78214d76a23cdd0b09863f60 [term 1 LEADER]: Becoming Leader. State: Replica: 9d6976ca78214d76a23cdd0b09863f60, State: Running, Role: LEADER
I20260929 19:01:10.370527 5799 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9d6976ca78214d76a23cdd0b09863f60 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 19:01:10.370551 5802 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9d6976ca78214d76a23cdd0b09863f60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d6976ca78214d76a23cdd0b09863f60" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41935 } }
I20260929 19:01:10.370919 5804 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d6976ca78214d76a23cdd0b09863f60 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9d6976ca78214d76a23cdd0b09863f60. Latest consensus state: current_term: 1 leader_uuid: "9d6976ca78214d76a23cdd0b09863f60" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d6976ca78214d76a23cdd0b09863f60" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41935 } } }
I20260929 19:01:10.370999 5804 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d6976ca78214d76a23cdd0b09863f60 [sys.catalog]: This master's current role is: LEADER
I20260929 19:01:10.371042 5803 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d6976ca78214d76a23cdd0b09863f60 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9d6976ca78214d76a23cdd0b09863f60" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d6976ca78214d76a23cdd0b09863f60" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41935 } } }
I20260929 19:01:10.371121 5803 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d6976ca78214d76a23cdd0b09863f60 [sys.catalog]: This master's current role is: LEADER
W20260929 19:01:10.372203 5818 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9d6976ca78214d76a23cdd0b09863f60: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 19:01:10.372265 5818 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 19:01:10.372315 5809 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 19:01:10.372460 5809 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 19:01:10.372879 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUbnLHE/build/release/bin/kudu as pid 5732
I20260929 19:01:10.372975 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790708447806542-392-0/minicluster-data/master-0/wal/instance
I20260929 19:01:10.373154 392 external_mini_cluster.cc:1428] Setting key 543fb531bbbbadd4793a2f0b8a40d009
I20260929 19:01:10.374086 5809 catalog_manager.cc:1347] Generated new cluster ID: 78d3d049feb74db0b1390e5a66837f95
I20260929 19:01:10.374137 5809 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 19:01:10.378790 5822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:10.373514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44704 (local address 127.0.98.62:41935)
0929 19:01:10.373704 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:10.373713 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:10.373747 (+ 34us) server_negotiation.cc:407] Connection header received
0929 19:01:10.373871 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:10.373877 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:10.373943 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:10.374069 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:01:10.374381 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:10.375673 (+ 1292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:10.376764 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:10.377374 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:10.377479 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:10.378128 (+ 649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:01:10.378131 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:01:10.378131 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 19:01:10.378322 (+ 191us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:01:10.378360 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 19:01:10.378373 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:01:10.378434 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0929 19:01:10.378543 (+ 109us) server_negotiation.cc:299] Negotiation successful
0929 19:01:10.378610 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":84,"threads_started":1}
I20260929 19:01:10.379849 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 19:01:10.392652 5809 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 19:01:10.393363 5809 catalog_manager.cc:1504] Loading token signing keys...
I20260929 19:01:10.405452 5809 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9d6976ca78214d76a23cdd0b09863f60: Generated new TSK 0
I20260929 19:01:10.405637 5809 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 19:01:10.421564 5822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:10.418008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44716 (local address 127.0.98.62:41935)
0929 19:01:10.418076 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:10.418078 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:10.418181 (+ 103us) server_negotiation.cc:407] Connection header received
0929 19:01:10.418249 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:10.418249 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:10.418260 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:10.418317 (+ 57us) server_negotiation.cc:226] Negotiated authn=JWT
0929 19:01:10.418454 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:10.418931 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:10.420061 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:10.420243 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:10.420298 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:10.420501 (+ 203us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 19:01:10.421337 (+ 836us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40707/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0929 19:01:10.421485 (+ 148us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:44716: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40707/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":41}
W20260929 19:01:10.421626 5822 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:44716: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40707/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260929 19:01:10.421631 5828 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:01:10.417950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41935 (local address 127.0.0.1:44716)
0929 19:01:10.418108 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:10.418119 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 19:01:10.418187 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:01:10.418325 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:01:10.418326 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:01:10.418328 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0929 19:01:10.418390 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:10.418393 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:10.419025 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:10.419026 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:10.419974 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:10.419977 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:10.420317 (+ 340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:10.420324 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:10.420383 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:10.420388 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 19:01:10.421440 (+ 1052us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40707/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0929 19:01:10.421529 (+ 89us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:41935: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40707/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":106,"threads_started":1}
I20260929 19:01:10.421890 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUbnLHE/build/release/bin/kudu with pid 5732
E20260929 19:01:10.432780 5829 webserver.cc:533] Webserver: sslize failed:
E20260929 19:01:10.432876 5829 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-29T19:01:10Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (218 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-29T19:01:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T19:01:10Z Disabled control of system clock
I20260929 19:01:10.465895 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUbnLHE/build/release/bin/kudu
/tmp/dist-test-taskUbnLHE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790708447806542-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790708447806542-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790708447806542-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790708447806542-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38657
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:43835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38657
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790708447806542-392-0/testchainca.cert with env {}
W20260929 19:01:10.542574 5833 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:01:10.542780 5833 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:01:10.542826 5833 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:01:10.543924 5833 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 19:01:10.543980 5833 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:01:10.544003 5833 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:01:10.544029 5833 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 19:01:10.544051 5833 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 19:01:10.544092 5833 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 19:01:10.544119 5833 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260929 19:01:10.545583 5833 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790708447806542-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790708447806542-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38657
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38657
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790708447806542-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790708447806542-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5833
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790708447806542-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 18:09:13 UTC on ccycloud.quasar-hpkalf.root.comops.site
build id 4680913
I20260929 19:01:10.545833 5833 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:01:10.546042 5833 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:01:10.550328 5841 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:01:10.550334 5839 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:01:10.550334 5838 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:01:10.550611 5833 server_base.cc:1034] running on GCE node
I20260929 19:01:10.550837 5833 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:01:10.551103 5833 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:01:10.552253 5833 hybrid_clock.cc:648] HybridClock initialized: now 1790708470552233 us; error 30 us; skew 500 ppm
I20260929 19:01:10.553359 5833 webserver.cc:460] Webserver started at http://127.0.98.62:43165/ using document root <none> and password file <none>
I20260929 19:01:10.553572 5833 fs_manager.cc:359] Metadata directory not provided
I20260929 19:01:10.553632 5833 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:01:10.553740 5833 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:01:10.554662 5833 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790708447806542-392-0/minicluster-data/master-0/data/instance:
uuid: "c9a60f278cc04eca9accc0d8e298a2af"
format_stamp: "Formatted at 2026-09-29 19:01:10 on dist-test-slave-lqkv"
server_key: "7af964ed1d48e27a05765e3b5322c09b"
server_key_iv: "ab2e60ef093559c03e0bf26eb16065d1"
server_key_version: "encryptionkey@0"
I20260929 19:01:10.555035 5833 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790708447806542-392-0/minicluster-data/master-0/wal/instance:
uuid: "c9a60f278cc04eca9accc0d8e298a2af"
format_stamp: "Formatted at 2026-09-29 19:01:10 on dist-test-slave-lqkv"
server_key: "7af964ed1d48e27a05765e3b5322c09b"
server_key_iv: "ab2e60ef093559c03e0bf26eb16065d1"
server_key_version: "encryptionkey@0"
I20260929 19:01:10.556209 5833 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 19:01:10.556898 5847 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:10.557085 5833 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 19:01:10.557170 5833 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790708447806542-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790708447806542-392-0/minicluster-data/master-0/wal
uuid: "c9a60f278cc04eca9accc0d8e298a2af"
format_stamp: "Formatted at 2026-09-29 19:01:10 on dist-test-slave-lqkv"
server_key: "7af964ed1d48e27a05765e3b5322c09b"
server_key_iv: "ab2e60ef093559c03e0bf26eb16065d1"
server_key_version: "encryptionkey@0"
I20260929 19:01:10.557247 5833 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790708447806542-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790708447806542-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790708447806542-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:01:10.587064 5833 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:01:10.587390 5833 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:01:10.587539 5833 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:01:10.591459 5833 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38657
I20260929 19:01:10.591495 5899 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38657 every 8 connection(s)
I20260929 19:01:10.591842 5833 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790708447806542-392-0/minicluster-data/master-0/data/info.pb
I20260929 19:01:10.592707 5900 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:01:10.594928 5900 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9a60f278cc04eca9accc0d8e298a2af: Bootstrap starting.
I20260929 19:01:10.595435 5900 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c9a60f278cc04eca9accc0d8e298a2af: Neither blocks nor log segments found. Creating new log.
I20260929 19:01:10.595665 5900 log.cc:824] T 00000000000000000000000000000000 P c9a60f278cc04eca9accc0d8e298a2af: Log is configured to *not* fsync() on all Append() calls
I20260929 19:01:10.596267 5900 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9a60f278cc04eca9accc0d8e298a2af: No bootstrap required, opened a new log
I20260929 19:01:10.597738 5900 raft_consensus.cc:348] T 00000000000000000000000000000000 P c9a60f278cc04eca9accc0d8e298a2af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9a60f278cc04eca9accc0d8e298a2af" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38657 } }
I20260929 19:01:10.597862 5900 raft_consensus.cc:374] T 00000000000000000000000000000000 P c9a60f278cc04eca9accc0d8e298a2af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:01:10.597882 5900 raft_consensus.cc:729] T 00000000000000000000000000000000 P c9a60f278cc04eca9accc0d8e298a2af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9a60f278cc04eca9accc0d8e298a2af, State: Initialized, Role: FOLLOWER
I20260929 19:01:10.597976 5900 consensus_queue.cc:260] T 00000000000000000000000000000000 P c9a60f278cc04eca9accc0d8e298a2af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9a60f278cc04eca9accc0d8e298a2af" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38657 } }
I20260929 19:01:10.598022 5900 raft_consensus.cc:388] T 00000000000000000000000000000000 P c9a60f278cc04eca9accc0d8e298a2af [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 19:01:10.598045 5900 raft_consensus.cc:482] T 00000000000000000000000000000000 P c9a60f278cc04eca9accc0d8e298a2af [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 19:01:10.598069 5900 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c9a60f278cc04eca9accc0d8e298a2af [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:01:10.598505 5900 raft_consensus.cc:504] T 00000000000000000000000000000000 P c9a60f278cc04eca9accc0d8e298a2af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9a60f278cc04eca9accc0d8e298a2af" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38657 } }
I20260929 19:01:10.598580 5900 leader_election.cc:302] T 00000000000000000000000000000000 P c9a60f278cc04eca9accc0d8e298a2af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c9a60f278cc04eca9accc0d8e298a2af; no voters:
I20260929 19:01:10.598688 5900 leader_election.cc:288] T 00000000000000000000000000000000 P c9a60f278cc04eca9accc0d8e298a2af [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 19:01:10.598752 5903 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c9a60f278cc04eca9accc0d8e298a2af [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:01:10.598884 5903 raft_consensus.cc:686] T 00000000000000000000000000000000 P c9a60f278cc04eca9accc0d8e298a2af [term 1 LEADER]: Becoming Leader. State: Replica: c9a60f278cc04eca9accc0d8e298a2af, State: Running, Role: LEADER
I20260929 19:01:10.598959 5900 sys_catalog.cc:564] T 00000000000000000000000000000000 P c9a60f278cc04eca9accc0d8e298a2af [sys.catalog]: configured and running, proceeding with master startup.
I20260929 19:01:10.598989 5903 consensus_queue.cc:237] T 00000000000000000000000000000000 P c9a60f278cc04eca9accc0d8e298a2af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9a60f278cc04eca9accc0d8e298a2af" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38657 } }
I20260929 19:01:10.599288 5904 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9a60f278cc04eca9accc0d8e298a2af [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c9a60f278cc04eca9accc0d8e298a2af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9a60f278cc04eca9accc0d8e298a2af" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38657 } } }
I20260929 19:01:10.599376 5904 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9a60f278cc04eca9accc0d8e298a2af [sys.catalog]: This master's current role is: LEADER
I20260929 19:01:10.599402 5905 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9a60f278cc04eca9accc0d8e298a2af [sys.catalog]: SysCatalogTable state changed. Reason: New leader c9a60f278cc04eca9accc0d8e298a2af. Latest consensus state: current_term: 1 leader_uuid: "c9a60f278cc04eca9accc0d8e298a2af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9a60f278cc04eca9accc0d8e298a2af" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38657 } } }
I20260929 19:01:10.599466 5905 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9a60f278cc04eca9accc0d8e298a2af [sys.catalog]: This master's current role is: LEADER
I20260929 19:01:10.600081 5909 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 19:01:10.600193 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUbnLHE/build/release/bin/kudu as pid 5833
I20260929 19:01:10.600301 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790708447806542-392-0/minicluster-data/master-0/wal/instance
I20260929 19:01:10.600495 5909 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 19:01:10.600510 392 external_mini_cluster.cc:1428] Setting key 50d34ec73762c8502f5c74117908eab1
I20260929 19:01:10.602138 5909 catalog_manager.cc:1347] Generated new cluster ID: fb3b93015e504a14ac54466d76602c98
I20260929 19:01:10.602187 5909 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 19:01:10.606462 5921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:10.600943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59936 (local address 127.0.98.62:38657)
0929 19:01:10.601325 (+ 382us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:10.601334 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:10.601434 (+ 100us) server_negotiation.cc:407] Connection header received
0929 19:01:10.601577 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:10.601583 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:10.601649 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:10.601782 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:01:10.601999 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:10.603316 (+ 1317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:10.604338 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:10.604990 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:10.605114 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:10.605744 (+ 630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:01:10.605748 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:01:10.605749 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 19:01:10.605947 (+ 198us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:01:10.606000 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 19:01:10.606013 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:01:10.606082 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0929 19:01:10.606147 (+ 65us) server_negotiation.cc:299] Negotiation successful
0929 19:01:10.606240 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":120,"threads_started":1}
I20260929 19:01:10.607306 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 19:01:10.624465 5909 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 19:01:10.625029 5909 catalog_manager.cc:1504] Loading token signing keys...
I20260929 19:01:10.631623 5909 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c9a60f278cc04eca9accc0d8e298a2af: Generated new TSK 0
I20260929 19:01:10.631803 5909 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 19:01:11.169824 5921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:10.636558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59940 (local address 127.0.98.62:38657)
0929 19:01:10.636619 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:10.636621 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:10.636743 (+ 122us) server_negotiation.cc:407] Connection header received
0929 19:01:10.636845 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:10.636846 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:10.636857 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:10.636930 (+ 73us) server_negotiation.cc:226] Negotiated authn=JWT
0929 19:01:10.637081 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:10.637576 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:10.638688 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:10.638879 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:10.638927 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:10.639116 (+ 189us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 19:01:11.169606 (+530490us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0929 19:01:11.169754 (+ 148us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:59940: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":38}
W20260929 19:01:11.169909 5921 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:59940: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260929 19:01:11.169955 5929 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:01:10.636498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38657 (local address 127.0.0.1:59940)
0929 19:01:10.636655 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:10.636679 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 19:01:10.636743 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:01:10.636940 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:01:10.636941 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:01:10.636943 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0929 19:01:10.637017 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:10.637018 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:10.637653 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:10.637654 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:10.638607 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:10.638609 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:10.638957 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:10.638963 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:10.639027 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:10.639032 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 19:01:11.169734 (+530702us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0929 19:01:11.169852 (+ 118us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:38657: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":109,"threads_started":1}
I20260929 19:01:11.170277 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUbnLHE/build/release/bin/kudu with pid 5833
2026-09-29T19:01:11Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (728 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (83 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (83 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708447806542-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 19:01:11 dist-test-slave-lqkv krb5kdc[5941](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 19:01:11 dist-test-slave-lqkv krb5kdc[5941](info): set up 2 sockets
Sep 29 19:01:11 dist-test-slave-lqkv krb5kdc[5941](info): commencing operation
krb5kdc: starting...
W20260929 19:01:12.364851 392 mini_kdc.cc:121] Time spent starting KDC: real 1.009s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708447806542-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708447806542-392-0/krb5kdc/test-user.keytab.
Sep 29 19:01:12 dist-test-slave-lqkv krb5kdc[5941](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708472, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T19:01:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T19:01:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708447806542-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708447806542-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 19:01:12.476074 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUbnLHE/build/release/bin/kudu
/tmp/dist-test-taskUbnLHE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708447806542-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708447806542-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708447806542-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708447806542-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35697
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708447806542-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35697
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708447806542-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708447806542-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708447806542-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:01:12.553081 5954 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:01:12.553282 5954 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:01:12.553316 5954 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:01:12.554435 5954 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 19:01:12.554497 5954 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:01:12.554523 5954 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:01:12.554553 5954 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 19:01:12.554577 5954 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 19:01:12.556160 5954 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708447806542-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708447806542-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35697
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708447806542-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35697
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708447806542-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5954
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708447806542-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 18:09:13 UTC on ccycloud.quasar-hpkalf.root.comops.site
build id 4680913
I20260929 19:01:12.556413 5954 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:01:12.556625 5954 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:01:12.561232 5962 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:01:12.561318 5954 server_base.cc:1034] running on GCE node
W20260929 19:01:12.561281 5959 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:01:12.561281 5960 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:01:12.561718 5954 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:01:12.562026 5954 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:01:12.563203 5954 hybrid_clock.cc:648] HybridClock initialized: now 1790708472563184 us; error 30 us; skew 500 ppm
Sep 29 19:01:12 dist-test-slave-lqkv krb5kdc[5941](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708472, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:01:12.568701 5954 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 19:01:12.569131 5954 webserver.cc:460] Webserver started at http://127.0.98.62:42583/ using document root <none> and password file <none>
I20260929 19:01:12.569357 5954 fs_manager.cc:359] Metadata directory not provided
I20260929 19:01:12.569422 5954 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:01:12.569566 5954 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:01:12.570655 5954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708447806542-392-0/minicluster-data/master-0/data/instance:
uuid: "31ee4c9e3f5345bbbac29749f14ff5bc"
format_stamp: "Formatted at 2026-09-29 19:01:12 on dist-test-slave-lqkv"
server_key: "8bfd314a15604ccdb757f2b99a7d0770"
server_key_iv: "c5962ab3ba9f6ffe5a5d3a373d912bd9"
server_key_version: "encryptionkey@0"
I20260929 19:01:12.571009 5954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708447806542-392-0/minicluster-data/master-0/wal/instance:
uuid: "31ee4c9e3f5345bbbac29749f14ff5bc"
format_stamp: "Formatted at 2026-09-29 19:01:12 on dist-test-slave-lqkv"
server_key: "8bfd314a15604ccdb757f2b99a7d0770"
server_key_iv: "c5962ab3ba9f6ffe5a5d3a373d912bd9"
server_key_version: "encryptionkey@0"
I20260929 19:01:12.572283 5954 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 19:01:12.573037 5969 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:12.573206 5954 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260929 19:01:12.573299 5954 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708447806542-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708447806542-392-0/minicluster-data/master-0/wal
uuid: "31ee4c9e3f5345bbbac29749f14ff5bc"
format_stamp: "Formatted at 2026-09-29 19:01:12 on dist-test-slave-lqkv"
server_key: "8bfd314a15604ccdb757f2b99a7d0770"
server_key_iv: "c5962ab3ba9f6ffe5a5d3a373d912bd9"
server_key_version: "encryptionkey@0"
I20260929 19:01:12.573376 5954 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708447806542-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708447806542-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708447806542-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:01:12.589115 5954 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:01:12.591396 5954 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:01:12.591550 5954 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:01:12.595329 5954 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35697
I20260929 19:01:12.595366 6021 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35697 every 8 connection(s)
I20260929 19:01:12.595855 5954 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708447806542-392-0/minicluster-data/master-0/data/info.pb
I20260929 19:01:12.596602 6022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:01:12.598829 6022 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31ee4c9e3f5345bbbac29749f14ff5bc: Bootstrap starting.
I20260929 19:01:12.599387 6022 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 31ee4c9e3f5345bbbac29749f14ff5bc: Neither blocks nor log segments found. Creating new log.
I20260929 19:01:12.599677 6022 log.cc:824] T 00000000000000000000000000000000 P 31ee4c9e3f5345bbbac29749f14ff5bc: Log is configured to *not* fsync() on all Append() calls
I20260929 19:01:12.600251 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUbnLHE/build/release/bin/kudu as pid 5954
I20260929 19:01:12.600363 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708447806542-392-0/minicluster-data/master-0/wal/instance
I20260929 19:01:12.600420 6022 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31ee4c9e3f5345bbbac29749f14ff5bc: No bootstrap required, opened a new log
I20260929 19:01:12.600557 392 external_mini_cluster.cc:1428] Setting key a1d71b603f4a66e79d7dd893b0572d5a
I20260929 19:01:12.602854 6022 raft_consensus.cc:348] T 00000000000000000000000000000000 P 31ee4c9e3f5345bbbac29749f14ff5bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31ee4c9e3f5345bbbac29749f14ff5bc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35697 } }
I20260929 19:01:12.602977 6022 raft_consensus.cc:374] T 00000000000000000000000000000000 P 31ee4c9e3f5345bbbac29749f14ff5bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:01:12.602999 6022 raft_consensus.cc:729] T 00000000000000000000000000000000 P 31ee4c9e3f5345bbbac29749f14ff5bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31ee4c9e3f5345bbbac29749f14ff5bc, State: Initialized, Role: FOLLOWER
I20260929 19:01:12.603134 6022 consensus_queue.cc:260] T 00000000000000000000000000000000 P 31ee4c9e3f5345bbbac29749f14ff5bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31ee4c9e3f5345bbbac29749f14ff5bc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35697 } }
I20260929 19:01:12.603196 6022 raft_consensus.cc:388] T 00000000000000000000000000000000 P 31ee4c9e3f5345bbbac29749f14ff5bc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 19:01:12.603252 6022 raft_consensus.cc:482] T 00000000000000000000000000000000 P 31ee4c9e3f5345bbbac29749f14ff5bc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 19:01:12.603300 6022 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 31ee4c9e3f5345bbbac29749f14ff5bc [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:01:12.603812 6022 raft_consensus.cc:504] T 00000000000000000000000000000000 P 31ee4c9e3f5345bbbac29749f14ff5bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31ee4c9e3f5345bbbac29749f14ff5bc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35697 } }
I20260929 19:01:12.603925 6022 leader_election.cc:302] T 00000000000000000000000000000000 P 31ee4c9e3f5345bbbac29749f14ff5bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 31ee4c9e3f5345bbbac29749f14ff5bc; no voters:
I20260929 19:01:12.604115 6022 leader_election.cc:288] T 00000000000000000000000000000000 P 31ee4c9e3f5345bbbac29749f14ff5bc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 19:01:12.604137 6027 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 31ee4c9e3f5345bbbac29749f14ff5bc [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:01:12.604244 6027 raft_consensus.cc:686] T 00000000000000000000000000000000 P 31ee4c9e3f5345bbbac29749f14ff5bc [term 1 LEADER]: Becoming Leader. State: Replica: 31ee4c9e3f5345bbbac29749f14ff5bc, State: Running, Role: LEADER
I20260929 19:01:12.604329 6027 consensus_queue.cc:237] T 00000000000000000000000000000000 P 31ee4c9e3f5345bbbac29749f14ff5bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31ee4c9e3f5345bbbac29749f14ff5bc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35697 } }
I20260929 19:01:12.604359 6022 sys_catalog.cc:564] T 00000000000000000000000000000000 P 31ee4c9e3f5345bbbac29749f14ff5bc [sys.catalog]: configured and running, proceeding with master startup.
I20260929 19:01:12.604676 6027 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31ee4c9e3f5345bbbac29749f14ff5bc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 31ee4c9e3f5345bbbac29749f14ff5bc. Latest consensus state: current_term: 1 leader_uuid: "31ee4c9e3f5345bbbac29749f14ff5bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31ee4c9e3f5345bbbac29749f14ff5bc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35697 } } }
I20260929 19:01:12.604760 6027 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31ee4c9e3f5345bbbac29749f14ff5bc [sys.catalog]: This master's current role is: LEADER
I20260929 19:01:12.604988 6032 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 19:01:12.605535 6032 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 19:01:12.605858 6028 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31ee4c9e3f5345bbbac29749f14ff5bc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "31ee4c9e3f5345bbbac29749f14ff5bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31ee4c9e3f5345bbbac29749f14ff5bc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35697 } } }
I20260929 19:01:12.606076 6028 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31ee4c9e3f5345bbbac29749f14ff5bc [sys.catalog]: This master's current role is: LEADER
I20260929 19:01:12.607259 6032 catalog_manager.cc:1347] Generated new cluster ID: b038e5ff16124717936ada7abf7aacd8
I20260929 19:01:12.607309 6032 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 29 19:01:12 dist-test-slave-lqkv krb5kdc[5941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708472, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 19:01:12.611399 6032 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 19:01:12.611943 6026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:12.601050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49734 (local address 127.0.98.62:35697)
0929 19:01:12.601213 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:12.601223 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:12.601254 (+ 31us) server_negotiation.cc:407] Connection header received
0929 19:01:12.601756 (+ 502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:12.601763 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:12.601807 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:12.601955 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:01:12.602566 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:12.603650 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:12.604643 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:12.605269 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:12.605379 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:12.608299 (+ 2920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:01:12.608303 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:01:12.608303 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:01:12.608313 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:01:12.610194 (+ 1881us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:12.610586 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:12.610595 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:12.610596 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:12.610692 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:12.610910 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:12.610911 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:12.610911 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:12.611158 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:01:12.611318 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0929 19:01:12.611426 (+ 108us) server_negotiation.cc:299] Negotiation successful
0929 19:01:12.611561 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":88,"threads_started":1}
I20260929 19:01:12.612129 6032 catalog_manager.cc:1504] Loading token signing keys...
I20260929 19:01:12.616276 6032 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 31ee4c9e3f5345bbbac29749f14ff5bc: Generated new TSK 0
I20260929 19:01:12.616418 6032 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708447806542-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708447806542-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260929 19:01:12.641193 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUbnLHE/build/release/bin/kudu
/tmp/dist-test-taskUbnLHE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708447806542-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708447806542-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708447806542-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708447806542-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35697
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708447806542-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708447806542-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708447806542-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708447806542-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:01:12.725368 6048 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:01:12.725582 6048 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:01:12.725637 6048 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:01:12.726799 6048 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:01:12.726862 6048 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:01:12.726905 6048 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 19:01:12.728359 6048 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708447806542-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708447806542-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708447806542-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708447806542-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35697
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6048
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708447806542-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 18:09:13 UTC on ccycloud.quasar-hpkalf.root.comops.site
build id 4680913
I20260929 19:01:12.728613 6048 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:01:12.728893 6048 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:01:12.733712 6056 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:01:12.733712 6053 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:01:12.733902 6054 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:01:12.734292 6048 server_base.cc:1034] running on GCE node
I20260929 19:01:12.734498 6048 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:01:12.734807 6048 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:01:12.735980 6048 hybrid_clock.cc:648] HybridClock initialized: now 1790708472735956 us; error 39 us; skew 500 ppm
Sep 29 19:01:12 dist-test-slave-lqkv krb5kdc[5941](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708472, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:01:12.738683 6048 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 19:01:12.739120 6048 webserver.cc:460] Webserver started at http://127.0.98.1:41063/ using document root <none> and password file <none>
I20260929 19:01:12.739347 6048 fs_manager.cc:359] Metadata directory not provided
I20260929 19:01:12.739411 6048 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:01:12.739537 6048 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:01:12.740522 6048 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708447806542-392-0/minicluster-data/ts-0/data/instance:
uuid: "08b24463373847ac9749b59b5a128e6f"
format_stamp: "Formatted at 2026-09-29 19:01:12 on dist-test-slave-lqkv"
server_key: "3efb5e1bc668fbb859332781d1f47251"
server_key_iv: "0ee19bf601be4e3dd27f80195dcafa74"
server_key_version: "encryptionkey@0"
I20260929 19:01:12.740945 6048 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708447806542-392-0/minicluster-data/ts-0/wal/instance:
uuid: "08b24463373847ac9749b59b5a128e6f"
format_stamp: "Formatted at 2026-09-29 19:01:12 on dist-test-slave-lqkv"
server_key: "3efb5e1bc668fbb859332781d1f47251"
server_key_iv: "0ee19bf601be4e3dd27f80195dcafa74"
server_key_version: "encryptionkey@0"
I20260929 19:01:12.742168 6048 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 19:01:12.742836 6063 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:12.742992 6048 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 19:01:12.743086 6048 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708447806542-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708447806542-392-0/minicluster-data/ts-0/wal
uuid: "08b24463373847ac9749b59b5a128e6f"
format_stamp: "Formatted at 2026-09-29 19:01:12 on dist-test-slave-lqkv"
server_key: "3efb5e1bc668fbb859332781d1f47251"
server_key_iv: "0ee19bf601be4e3dd27f80195dcafa74"
server_key_version: "encryptionkey@0"
I20260929 19:01:12.743184 6048 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708447806542-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708447806542-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708447806542-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:01:12.760701 6048 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:01:12.762720 6048 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:01:12.762876 6048 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:01:12.763134 6048 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:01:12.763447 6048 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:01:12.763501 6048 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:12.763543 6048 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:01:12.763569 6048 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:12.769330 6048 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40617
I20260929 19:01:12.769369 6176 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40617 every 8 connection(s)
I20260929 19:01:12.769658 6048 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708447806542-392-0/minicluster-data/ts-0/data/info.pb
Sep 29 19:01:12 dist-test-slave-lqkv krb5kdc[5941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708472, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 19:01:12.775660 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUbnLHE/build/release/bin/kudu as pid 6048
I20260929 19:01:12.775792 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708447806542-392-0/minicluster-data/ts-0/wal/instance
I20260929 19:01:12.775995 392 external_mini_cluster.cc:1428] Setting key 14d17431ec42d19273190dabfbde587b
I20260929 19:01:12.779338 6026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:12.770303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36003 (local address 127.0.98.62:35697)
0929 19:01:12.770360 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:12.770362 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:12.770594 (+ 232us) server_negotiation.cc:407] Connection header received
0929 19:01:12.771503 (+ 909us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:12.771504 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:12.771514 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:12.771568 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:01:12.772138 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:12.772636 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:12.773780 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:12.773969 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:12.774023 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:12.776221 (+ 2198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:01:12.776222 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:01:12.776223 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:01:12.776229 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:01:12.777907 (+ 1678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:12.778304 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:12.778305 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:12.778306 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:12.778372 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:12.778647 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:12.778649 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:12.778649 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:12.778766 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:01:12.778845 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0929 19:01:12.779162 (+ 317us) server_negotiation.cc:299] Negotiation successful
0929 19:01:12.779254 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260929 19:01:12.779498 6179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:12.770292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35697 (local address 127.0.98.1:36003)
0929 19:01:12.770442 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:12.770482 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 19:01:12.771417 (+ 935us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:01:12.771626 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:01:12.771629 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:01:12.771871 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:01:12.772066 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:12.772070 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:12.772743 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:12.772744 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:12.773687 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:12.773689 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:12.774040 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:12.774041 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:12.774146 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:12.774582 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:01:12.774590 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:01:12.776108 (+ 1518us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:01:12.777986 (+ 1878us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:01:12.777992 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:01:12.777993 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:01:12.778207 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:01:12.778439 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:01:12.778440 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:01:12.778440 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:01:12.778553 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:01:12.778834 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:01:12.778840 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:01:12.778992 (+ 152us) client_negotiation.cc:769] Sending connection context
0929 19:01:12.779135 (+ 143us) client_negotiation.cc:240] Negotiation successful
0929 19:01:12.779278 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":82,"threads_started":1}
I20260929 19:01:12.779932 6177 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35697
I20260929 19:01:12.780053 6177 heartbeater.cc:464] Registering TS with master...
I20260929 19:01:12.780241 6177 heartbeater.cc:511] Master 127.0.98.62:35697 requested a full tablet report, sending...
I20260929 19:01:12.780758 5986 ts_manager.cc:194] Registered new tserver with Master: 08b24463373847ac9749b59b5a128e6f (127.0.98.1:40617)
I20260929 19:01:12.781711 5986 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:36003
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708447806542-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708447806542-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260929 19:01:12.801350 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUbnLHE/build/release/bin/kudu
/tmp/dist-test-taskUbnLHE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708447806542-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708447806542-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708447806542-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708447806542-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35697
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708447806542-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708447806542-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708447806542-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708447806542-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:01:12.877796 6182 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:01:12.877969 6182 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:01:12.877990 6182 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:01:12.879060 6182 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:01:12.879091 6182 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:01:12.879115 6182 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 19:01:12.880420 6182 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708447806542-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708447806542-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708447806542-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708447806542-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35697
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6182
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708447806542-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 18:09:13 UTC on ccycloud.quasar-hpkalf.root.comops.site
build id 4680913
I20260929 19:01:12.880581 6182 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:01:12.880734 6182 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:01:12.885367 6190 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:01:12.885387 6182 server_base.cc:1034] running on GCE node
W20260929 19:01:12.885361 6188 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:01:12.885361 6187 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:01:12.885797 6182 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:01:12.886014 6182 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:01:12.887166 6182 hybrid_clock.cc:648] HybridClock initialized: now 1790708472887159 us; error 34 us; skew 500 ppm
Sep 29 19:01:12 dist-test-slave-lqkv krb5kdc[5941](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708472, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:01:12.889460 6182 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 19:01:12.889861 6182 webserver.cc:460] Webserver started at http://127.0.98.2:40813/ using document root <none> and password file <none>
I20260929 19:01:12.890080 6182 fs_manager.cc:359] Metadata directory not provided
I20260929 19:01:12.890146 6182 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:01:12.890297 6182 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:01:12.891283 6182 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708447806542-392-0/minicluster-data/ts-1/data/instance:
uuid: "7f941942c3b2481d922de20b7f982053"
format_stamp: "Formatted at 2026-09-29 19:01:12 on dist-test-slave-lqkv"
server_key: "3108653f0f64a876e0fb6ec082d31fe6"
server_key_iv: "96c44d8e5cf88ffc25229687552548a3"
server_key_version: "encryptionkey@0"
I20260929 19:01:12.891634 6182 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708447806542-392-0/minicluster-data/ts-1/wal/instance:
uuid: "7f941942c3b2481d922de20b7f982053"
format_stamp: "Formatted at 2026-09-29 19:01:12 on dist-test-slave-lqkv"
server_key: "3108653f0f64a876e0fb6ec082d31fe6"
server_key_iv: "96c44d8e5cf88ffc25229687552548a3"
server_key_version: "encryptionkey@0"
I20260929 19:01:12.892971 6182 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260929 19:01:12.893669 6197 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:12.893839 6182 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 19:01:12.893930 6182 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708447806542-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708447806542-392-0/minicluster-data/ts-1/wal
uuid: "7f941942c3b2481d922de20b7f982053"
format_stamp: "Formatted at 2026-09-29 19:01:12 on dist-test-slave-lqkv"
server_key: "3108653f0f64a876e0fb6ec082d31fe6"
server_key_iv: "96c44d8e5cf88ffc25229687552548a3"
server_key_version: "encryptionkey@0"
I20260929 19:01:12.894012 6182 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708447806542-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708447806542-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708447806542-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:01:12.904136 6182 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:01:12.905997 6182 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:01:12.906155 6182 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:01:12.906361 6182 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:01:12.906628 6182 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:01:12.906697 6182 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:12.906737 6182 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:01:12.906764 6182 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:12.912122 6182 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46553
I20260929 19:01:12.912181 6310 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46553 every 8 connection(s)
I20260929 19:01:12.912474 6182 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708447806542-392-0/minicluster-data/ts-1/data/info.pb
I20260929 19:01:12.915249 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUbnLHE/build/release/bin/kudu as pid 6182
I20260929 19:01:12.915365 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708447806542-392-0/minicluster-data/ts-1/wal/instance
I20260929 19:01:12.915547 392 external_mini_cluster.cc:1428] Setting key 1b224f15254e825ccad144eaa8f935cc
Sep 29 19:01:12 dist-test-slave-lqkv krb5kdc[5941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708472, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 19:01:12.921851 6026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:12.913149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35691 (local address 127.0.98.62:35697)
0929 19:01:12.913189 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:12.913191 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:12.913375 (+ 184us) server_negotiation.cc:407] Connection header received
0929 19:01:12.914164 (+ 789us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:12.914165 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:12.914174 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:12.914222 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:01:12.914732 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:12.915214 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:12.916569 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:12.916796 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:12.916897 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:12.918983 (+ 2086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:01:12.918984 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:01:12.918984 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:01:12.918990 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:01:12.920451 (+ 1461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:12.920961 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:12.920962 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:12.920963 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:12.921010 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:12.921292 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:12.921292 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:12.921293 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:12.921386 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:01:12.921432 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0929 19:01:12.921735 (+ 303us) server_negotiation.cc:299] Negotiation successful
0929 19:01:12.921788 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260929 19:01:12.922091 6313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:12.913113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35697 (local address 127.0.98.2:35691)
0929 19:01:12.913251 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:12.913287 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0929 19:01:12.914091 (+ 804us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:01:12.914267 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:01:12.914270 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:01:12.914509 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:01:12.914669 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:12.914672 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:12.915311 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:12.915312 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:12.916488 (+ 1176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:12.916490 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:12.916913 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:12.916914 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:12.917034 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:12.917497 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:01:12.917504 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:01:12.918870 (+ 1366us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:01:12.920533 (+ 1663us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:01:12.920539 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:01:12.920541 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:01:12.920886 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:01:12.921076 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:01:12.921078 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:01:12.921078 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:01:12.921216 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:01:12.921449 (+ 233us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:01:12.921455 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:01:12.921605 (+ 150us) client_negotiation.cc:769] Sending connection context
0929 19:01:12.921717 (+ 112us) client_negotiation.cc:240] Negotiation successful
0929 19:01:12.921881 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":80,"threads_started":1}
I20260929 19:01:12.922592 6311 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35697
I20260929 19:01:12.922674 6311 heartbeater.cc:464] Registering TS with master...
I20260929 19:01:12.922849 6311 heartbeater.cc:511] Master 127.0.98.62:35697 requested a full tablet report, sending...
I20260929 19:01:12.923197 5986 ts_manager.cc:194] Registered new tserver with Master: 7f941942c3b2481d922de20b7f982053 (127.0.98.2:46553)
I20260929 19:01:12.923511 5986 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:35691
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708447806542-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708447806542-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260929 19:01:12.939595 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUbnLHE/build/release/bin/kudu
/tmp/dist-test-taskUbnLHE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708447806542-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708447806542-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708447806542-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708447806542-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35697
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708447806542-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708447806542-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708447806542-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708447806542-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:01:13.017987 6316 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:01:13.018167 6316 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:01:13.018188 6316 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:01:13.019336 6316 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:01:13.019375 6316 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:01:13.019418 6316 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 19:01:13.020915 6316 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708447806542-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708447806542-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708447806542-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708447806542-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35697
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6316
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708447806542-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 18:09:13 UTC on ccycloud.quasar-hpkalf.root.comops.site
build id 4680913
I20260929 19:01:13.021134 6316 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:01:13.021342 6316 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:01:13.025944 6321 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:01:13.025945 6322 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:01:13.026007 6324 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:01:13.026104 6316 server_base.cc:1034] running on GCE node
I20260929 19:01:13.026307 6316 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:01:13.026582 6316 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:01:13.027740 6316 hybrid_clock.cc:648] HybridClock initialized: now 1790708473027714 us; error 36 us; skew 500 ppm
Sep 29 19:01:13 dist-test-slave-lqkv krb5kdc[5941](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708473, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:01:13.030275 6316 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 19:01:13.030684 6316 webserver.cc:460] Webserver started at http://127.0.98.3:39001/ using document root <none> and password file <none>
I20260929 19:01:13.030906 6316 fs_manager.cc:359] Metadata directory not provided
I20260929 19:01:13.030972 6316 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:01:13.031098 6316 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:01:13.032049 6316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708447806542-392-0/minicluster-data/ts-2/data/instance:
uuid: "40b4c2c5c151476fbee2088add033881"
format_stamp: "Formatted at 2026-09-29 19:01:13 on dist-test-slave-lqkv"
server_key: "9cbaa88b98fbf9a6b16f240483179688"
server_key_iv: "30f7f12f012a72904282c6ec8360ed5e"
server_key_version: "encryptionkey@0"
I20260929 19:01:13.032425 6316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708447806542-392-0/minicluster-data/ts-2/wal/instance:
uuid: "40b4c2c5c151476fbee2088add033881"
format_stamp: "Formatted at 2026-09-29 19:01:13 on dist-test-slave-lqkv"
server_key: "9cbaa88b98fbf9a6b16f240483179688"
server_key_iv: "30f7f12f012a72904282c6ec8360ed5e"
server_key_version: "encryptionkey@0"
I20260929 19:01:13.033763 6316 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 19:01:13.034435 6331 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:13.034605 6316 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 19:01:13.034698 6316 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708447806542-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708447806542-392-0/minicluster-data/ts-2/wal
uuid: "40b4c2c5c151476fbee2088add033881"
format_stamp: "Formatted at 2026-09-29 19:01:13 on dist-test-slave-lqkv"
server_key: "9cbaa88b98fbf9a6b16f240483179688"
server_key_iv: "30f7f12f012a72904282c6ec8360ed5e"
server_key_version: "encryptionkey@0"
I20260929 19:01:13.034780 6316 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708447806542-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708447806542-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708447806542-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:01:13.047267 6316 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:01:13.049464 6316 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:01:13.049618 6316 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:01:13.049814 6316 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:01:13.050107 6316 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:01:13.050163 6316 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:13.050199 6316 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:01:13.050225 6316 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:13.055721 6316 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39373
I20260929 19:01:13.055763 6444 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39373 every 8 connection(s)
I20260929 19:01:13.056057 6316 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708447806542-392-0/minicluster-data/ts-2/data/info.pb
Sep 29 19:01:13 dist-test-slave-lqkv krb5kdc[5941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708473, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 19:01:13.064240 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUbnLHE/build/release/bin/kudu as pid 6316
I20260929 19:01:13.064342 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708447806542-392-0/minicluster-data/ts-2/wal/instance
I20260929 19:01:13.064536 392 external_mini_cluster.cc:1428] Setting key b69082a1b2d1d38c9b450e2ea93dbca2
I20260929 19:01:13.064530 6026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:13.056797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40123 (local address 127.0.98.62:35697)
0929 19:01:13.056928 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:13.056930 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:13.057079 (+ 149us) server_negotiation.cc:407] Connection header received
0929 19:01:13.057937 (+ 858us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:13.057938 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:13.057948 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:13.058003 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:01:13.058528 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:13.059003 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:13.059902 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:13.060076 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:13.060124 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:13.061982 (+ 1858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:01:13.061983 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:01:13.061984 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:01:13.061989 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:01:13.063415 (+ 1426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:13.063726 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:13.063726 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:13.063726 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:13.063772 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:13.063960 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:13.063960 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:13.063961 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:13.064071 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:01:13.064144 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 19:01:13.064365 (+ 221us) server_negotiation.cc:299] Negotiation successful
0929 19:01:13.064439 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
I20260929 19:01:13.064710 6447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:13.056762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35697 (local address 127.0.98.3:40123)
0929 19:01:13.056962 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:13.057000 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 19:01:13.057857 (+ 857us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:01:13.058062 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:01:13.058066 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:01:13.058288 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:01:13.058464 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:13.058467 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:13.059073 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:13.059073 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:13.059831 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:13.059833 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:13.060141 (+ 308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:13.060142 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:13.060239 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:13.060619 (+ 380us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:01:13.060626 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:01:13.061908 (+ 1282us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:01:13.063480 (+ 1572us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:01:13.063483 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:01:13.063485 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:01:13.063667 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:01:13.063826 (+ 159us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:01:13.063826 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:01:13.063826 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:01:13.063909 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:01:13.064137 (+ 228us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:01:13.064140 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:01:13.064235 (+ 95us) client_negotiation.cc:769] Sending connection context
0929 19:01:13.064337 (+ 102us) client_negotiation.cc:240] Negotiation successful
0929 19:01:13.064488 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":139,"threads_started":1}
I20260929 19:01:13.065263 6445 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35697
I20260929 19:01:13.065376 6445 heartbeater.cc:464] Registering TS with master...
I20260929 19:01:13.065552 6445 heartbeater.cc:511] Master 127.0.98.62:35697 requested a full tablet report, sending...
I20260929 19:01:13.065876 5986 ts_manager.cc:194] Registered new tserver with Master: 40b4c2c5c151476fbee2088add033881 (127.0.98.3:39373)
I20260929 19:01:13.066233 5986 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:40123
I20260929 19:01:13.076251 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708447806542-392-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708447806542-392-0/krb5kdc/test-admin.keytab.
Sep 29 19:01:13 dist-test-slave-lqkv krb5kdc[5941](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708473, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:01:13.090408 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260929 19:01:13.091981 6452 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:01:13.091239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52808 (local address 127.0.98.1:40617)
0929 19:01:13.091401 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:13.091406 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:13.091416 (+ 10us) server_negotiation.cc:407] Connection header received
0929 19:01:13.091441 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:13.091443 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:13.091456 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:13.091578 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:01:13.091898 (+ 320us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52808: BlockingRecv error: recv got EOF from 127.0.0.1:52808 (error 108)
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":89,"threads_started":1}
W20260929 19:01:13.092203 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708447806542-392-0/krb5kdc/krb5cc' of type FILE
Sep 29 19:01:13 dist-test-slave-lqkv krb5kdc[5941](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708473, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:01:13.093246 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 29 19:01:13 dist-test-slave-lqkv krb5kdc[5941](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708473, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:01:13.094193 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 29 19:01:13 dist-test-slave-lqkv krb5kdc[5941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708473, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260929 19:01:13.101969 6452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:13.094819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52824 (local address 127.0.98.1:40617)
0929 19:01:13.094900 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:13.094902 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:13.094934 (+ 32us) server_negotiation.cc:407] Connection header received
0929 19:01:13.094987 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:13.094988 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:13.094997 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:13.095043 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:01:13.095416 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:13.096120 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:13.096976 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:13.097286 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:13.097341 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:13.099389 (+ 2048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:01:13.099397 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:01:13.099399 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:01:13.099410 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:01:13.100957 (+ 1547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:13.101244 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:13.101246 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:13.101246 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:13.101297 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:13.101456 (+ 159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:13.101457 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:13.101457 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:13.101574 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:01:13.101669 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0929 19:01:13.101741 (+ 72us) server_negotiation.cc:299] Negotiation successful
0929 19:01:13.101798 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260929 19:01:13.103783 6452 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:01:13.103108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52834 (local address 127.0.98.1:40617)
0929 19:01:13.103159 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:13.103160 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:13.103201 (+ 41us) server_negotiation.cc:407] Connection header received
0929 19:01:13.103246 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:13.103246 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:13.103255 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:13.103301 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:01:13.103732 (+ 431us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52834: BlockingRecv error: recv got EOF from 127.0.0.1:52834 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
W20260929 19:01:13.103875 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708447806542-392-0/krb5kdc/krb5cc' of type FILE
Sep 29 19:01:13 dist-test-slave-lqkv krb5kdc[5941](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708473, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:01:13.104964 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 29 19:01:13 dist-test-slave-lqkv krb5kdc[5941](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708473, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:01:13.105993 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 29 19:01:13 dist-test-slave-lqkv krb5kdc[5941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708473, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260929 19:01:13.112391 6452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:13.106496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52844 (local address 127.0.98.1:40617)
0929 19:01:13.106548 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:13.106549 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:13.106576 (+ 27us) server_negotiation.cc:407] Connection header received
0929 19:01:13.106629 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:13.106630 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:13.106636 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:13.106672 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:01:13.106999 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:13.107462 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:13.108102 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:13.108300 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:13.108348 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:13.110238 (+ 1890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:01:13.110239 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:01:13.110239 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:01:13.110245 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:01:13.111391 (+ 1146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:13.111691 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:13.111692 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:13.111692 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:13.111754 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:13.111925 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:13.111925 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:13.111926 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:13.112010 (+ 84us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:01:13.112080 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 19:01:13.112192 (+ 112us) server_negotiation.cc:299] Negotiation successful
0929 19:01:13.112268 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
W20260929 19:01:13.113932 6452 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:01:13.113335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52854 (local address 127.0.98.1:40617)
0929 19:01:13.113377 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:13.113379 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:13.113422 (+ 43us) server_negotiation.cc:407] Connection header received
0929 19:01:13.113490 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:13.113490 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:13.113499 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:13.113552 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:01:13.113886 (+ 334us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52854: BlockingRecv error: recv got EOF from 127.0.0.1:52854 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260929 19:01:13.114017 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790708447806542-392-0/krb5kdc/krb5cc' of type FILE
Sep 29 19:01:13 dist-test-slave-lqkv krb5kdc[5941](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708473, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:01:13.114910 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 29 19:01:13 dist-test-slave-lqkv krb5kdc[5941](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708473, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:01:13.115695 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 29 19:01:13 dist-test-slave-lqkv krb5kdc[5941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708473, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260929 19:01:13.121922 6452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:13.116187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52858 (local address 127.0.98.1:40617)
0929 19:01:13.116243 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:13.116245 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:13.116425 (+ 180us) server_negotiation.cc:407] Connection header received
0929 19:01:13.116452 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:13.116453 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:13.116460 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:13.116506 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:01:13.116912 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:13.117328 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:13.117928 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:13.118087 (+ 159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:13.118132 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:13.119923 (+ 1791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:01:13.119924 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:01:13.119925 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:01:13.119930 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:01:13.121094 (+ 1164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:13.121371 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:13.121372 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:13.121372 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:13.121423 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:13.121562 (+ 139us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:13.121563 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:13.121563 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:13.121631 (+ 68us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:01:13.121689 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0929 19:01:13.121769 (+ 80us) server_negotiation.cc:299] Negotiation successful
0929 19:01:13.121816 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260929 19:01:13.122488 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUbnLHE/build/release/bin/kudu with pid 6048
I20260929 19:01:13.127512 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUbnLHE/build/release/bin/kudu with pid 6182
I20260929 19:01:13.131488 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUbnLHE/build/release/bin/kudu with pid 6316
I20260929 19:01:13.135341 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUbnLHE/build/release/bin/kudu with pid 5954
2026-09-29T19:01:13Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (1798 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 19:01:13 dist-test-slave-lqkv krb5kdc[6465](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 19:01:13 dist-test-slave-lqkv krb5kdc[6465](info): set up 2 sockets
Sep 29 19:01:13 dist-test-slave-lqkv krb5kdc[6465](info): commencing operation
krb5kdc: starting...
W20260929 19:01:14.169138 392 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/krb5kdc/test-user.keytab.
Sep 29 19:01:14 dist-test-slave-lqkv krb5kdc[6465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708474, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T19:01:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T19:01:14Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260929 19:01:14.277316 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUbnLHE/build/release/bin/kudu
/tmp/dist-test-taskUbnLHE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42245
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42245
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:01:14.353508 6478 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:01:14.353724 6478 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:01:14.353760 6478 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:01:14.354887 6478 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 19:01:14.354954 6478 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:01:14.354978 6478 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:01:14.355000 6478 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 19:01:14.355036 6478 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 19:01:14.356478 6478 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42245
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42245
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6478
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 18:09:13 UTC on ccycloud.quasar-hpkalf.root.comops.site
build id 4680913
I20260929 19:01:14.356741 6478 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:01:14.356978 6478 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:01:14.361570 6484 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:01:14.361565 6483 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:01:14.361733 6478 server_base.cc:1034] running on GCE node
W20260929 19:01:14.361546 6486 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:01:14.362006 6478 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:01:14.362286 6478 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:01:14.363449 6478 hybrid_clock.cc:648] HybridClock initialized: now 1790708474363444 us; error 34 us; skew 500 ppm
Sep 29 19:01:14 dist-test-slave-lqkv krb5kdc[6465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708474, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:01:14.365799 6478 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260929 19:01:14.366212 6478 webserver.cc:460] Webserver started at http://127.0.98.62:46043/ using document root <none> and password file <none>
I20260929 19:01:14.366436 6478 fs_manager.cc:359] Metadata directory not provided
I20260929 19:01:14.366500 6478 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:01:14.366650 6478 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:01:14.367640 6478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/minicluster-data/master-0/data/instance:
uuid: "ea566150fd024ed9b9485efbb8ee914c"
format_stamp: "Formatted at 2026-09-29 19:01:14 on dist-test-slave-lqkv"
server_key: "97e1604a7720ae25c52b1127c15de20d"
server_key_iv: "4821ee5b4f9cf43cee99763000d3bf65"
server_key_version: "encryptionkey@0"
I20260929 19:01:14.367982 6478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/minicluster-data/master-0/wal/instance:
uuid: "ea566150fd024ed9b9485efbb8ee914c"
format_stamp: "Formatted at 2026-09-29 19:01:14 on dist-test-slave-lqkv"
server_key: "97e1604a7720ae25c52b1127c15de20d"
server_key_iv: "4821ee5b4f9cf43cee99763000d3bf65"
server_key_version: "encryptionkey@0"
I20260929 19:01:14.369290 6478 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 19:01:14.369936 6493 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:14.370106 6478 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 19:01:14.370206 6478 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/minicluster-data/master-0/wal
uuid: "ea566150fd024ed9b9485efbb8ee914c"
format_stamp: "Formatted at 2026-09-29 19:01:14 on dist-test-slave-lqkv"
server_key: "97e1604a7720ae25c52b1127c15de20d"
server_key_iv: "4821ee5b4f9cf43cee99763000d3bf65"
server_key_version: "encryptionkey@0"
I20260929 19:01:14.370275 6478 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:01:14.390267 6478 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:01:14.391930 6478 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:01:14.392091 6478 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:01:14.395704 6478 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42245
I20260929 19:01:14.395743 6545 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42245 every 8 connection(s)
I20260929 19:01:14.396049 6478 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/minicluster-data/master-0/data/info.pb
I20260929 19:01:14.396986 6546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:01:14.399008 6546 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea566150fd024ed9b9485efbb8ee914c: Bootstrap starting.
I20260929 19:01:14.399484 6546 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ea566150fd024ed9b9485efbb8ee914c: Neither blocks nor log segments found. Creating new log.
I20260929 19:01:14.399714 6546 log.cc:824] T 00000000000000000000000000000000 P ea566150fd024ed9b9485efbb8ee914c: Log is configured to *not* fsync() on all Append() calls
I20260929 19:01:14.400295 6546 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea566150fd024ed9b9485efbb8ee914c: No bootstrap required, opened a new log
I20260929 19:01:14.401223 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUbnLHE/build/release/bin/kudu as pid 6478
I20260929 19:01:14.401350 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/minicluster-data/master-0/wal/instance
I20260929 19:01:14.401525 392 external_mini_cluster.cc:1428] Setting key bdcb4a605d0a840fef013b0deb77c827
I20260929 19:01:14.402016 6546 raft_consensus.cc:348] T 00000000000000000000000000000000 P ea566150fd024ed9b9485efbb8ee914c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea566150fd024ed9b9485efbb8ee914c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42245 } }
I20260929 19:01:14.402240 6546 raft_consensus.cc:374] T 00000000000000000000000000000000 P ea566150fd024ed9b9485efbb8ee914c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:01:14.402319 6546 raft_consensus.cc:729] T 00000000000000000000000000000000 P ea566150fd024ed9b9485efbb8ee914c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea566150fd024ed9b9485efbb8ee914c, State: Initialized, Role: FOLLOWER
I20260929 19:01:14.402482 6546 consensus_queue.cc:260] T 00000000000000000000000000000000 P ea566150fd024ed9b9485efbb8ee914c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea566150fd024ed9b9485efbb8ee914c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42245 } }
I20260929 19:01:14.402604 6546 raft_consensus.cc:388] T 00000000000000000000000000000000 P ea566150fd024ed9b9485efbb8ee914c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 19:01:14.402665 6546 raft_consensus.cc:482] T 00000000000000000000000000000000 P ea566150fd024ed9b9485efbb8ee914c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 19:01:14.402725 6546 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ea566150fd024ed9b9485efbb8ee914c [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:01:14.403743 6546 raft_consensus.cc:504] T 00000000000000000000000000000000 P ea566150fd024ed9b9485efbb8ee914c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea566150fd024ed9b9485efbb8ee914c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42245 } }
I20260929 19:01:14.403861 6546 leader_election.cc:302] T 00000000000000000000000000000000 P ea566150fd024ed9b9485efbb8ee914c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ea566150fd024ed9b9485efbb8ee914c; no voters:
I20260929 19:01:14.404027 6546 leader_election.cc:288] T 00000000000000000000000000000000 P ea566150fd024ed9b9485efbb8ee914c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 19:01:14.404070 6551 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ea566150fd024ed9b9485efbb8ee914c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:01:14.404209 6551 raft_consensus.cc:686] T 00000000000000000000000000000000 P ea566150fd024ed9b9485efbb8ee914c [term 1 LEADER]: Becoming Leader. State: Replica: ea566150fd024ed9b9485efbb8ee914c, State: Running, Role: LEADER
I20260929 19:01:14.404271 6546 sys_catalog.cc:564] T 00000000000000000000000000000000 P ea566150fd024ed9b9485efbb8ee914c [sys.catalog]: configured and running, proceeding with master startup.
I20260929 19:01:14.404947 6551 consensus_queue.cc:237] T 00000000000000000000000000000000 P ea566150fd024ed9b9485efbb8ee914c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea566150fd024ed9b9485efbb8ee914c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42245 } }
I20260929 19:01:14.405498 6552 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea566150fd024ed9b9485efbb8ee914c [sys.catalog]: SysCatalogTable state changed. Reason: New leader ea566150fd024ed9b9485efbb8ee914c. Latest consensus state: current_term: 1 leader_uuid: "ea566150fd024ed9b9485efbb8ee914c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea566150fd024ed9b9485efbb8ee914c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42245 } } }
I20260929 19:01:14.405616 6552 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea566150fd024ed9b9485efbb8ee914c [sys.catalog]: This master's current role is: LEADER
I20260929 19:01:14.405766 6553 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea566150fd024ed9b9485efbb8ee914c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ea566150fd024ed9b9485efbb8ee914c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea566150fd024ed9b9485efbb8ee914c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42245 } } }
W20260929 19:01:14.406044 6565 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ea566150fd024ed9b9485efbb8ee914c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 19:01:14.406822 6565 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 19:01:14.406883 6553 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea566150fd024ed9b9485efbb8ee914c [sys.catalog]: This master's current role is: LEADER
Sep 29 19:01:14 dist-test-slave-lqkv krb5kdc[6465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708474, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260929 19:01:14.411900 6550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:14.402033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54544 (local address 127.0.98.62:42245)
0929 19:01:14.402199 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:14.402239 (+ 40us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:14.402271 (+ 32us) server_negotiation.cc:407] Connection header received
0929 19:01:14.402399 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:14.402406 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:14.402476 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:14.402603 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:01:14.403435 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:14.404541 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:14.405576 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:14.406155 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:14.406276 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:14.408789 (+ 2513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:01:14.408792 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:01:14.408792 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:01:14.408837 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:01:14.410447 (+ 1610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:14.410829 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:14.410831 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:14.410832 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:14.410926 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:14.411122 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:14.411122 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:14.411123 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:14.411329 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:01:14.411438 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 19:01:14.411575 (+ 137us) server_negotiation.cc:299] Negotiation successful
0929 19:01:14.411665 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":95,"threads_started":1}
I20260929 19:01:14.416917 6568 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 19:01:14.417121 6568 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 19:01:14.418463 6568 catalog_manager.cc:1347] Generated new cluster ID: 273b8d19129f4d25a0e7c884d5c7d851
I20260929 19:01:14.418514 6568 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 19:01:14.425673 6568 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 19:01:14.426316 6568 catalog_manager.cc:1504] Loading token signing keys...
I20260929 19:01:14.433032 6568 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ea566150fd024ed9b9485efbb8ee914c: Generated new TSK 0
I20260929 19:01:14.433182 6568 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260929 19:01:14.463479 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUbnLHE/build/release/bin/kudu
/tmp/dist-test-taskUbnLHE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42245
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:01:14.553319 6572 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:01:14.553501 6572 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:01:14.553567 6572 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:01:14.554777 6572 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:01:14.554819 6572 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:01:14.554845 6572 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 19:01:14.556250 6572 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42245
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6572
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 18:09:13 UTC on ccycloud.quasar-hpkalf.root.comops.site
build id 4680913
I20260929 19:01:14.556468 6572 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:01:14.556705 6572 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 19:01:14.561496 6572 server_base.cc:1034] running on GCE node
W20260929 19:01:14.561476 6577 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:01:14.561468 6578 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:01:14.561476 6580 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:01:14.561863 6572 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:01:14.562119 6572 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:01:14.563323 6572 hybrid_clock.cc:648] HybridClock initialized: now 1790708474563295 us; error 38 us; skew 500 ppm
Sep 29 19:01:14 dist-test-slave-lqkv krb5kdc[6465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708474, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:01:14.565693 6572 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260929 19:01:14.566087 6572 webserver.cc:460] Webserver started at http://127.0.98.1:33741/ using document root <none> and password file <none>
I20260929 19:01:14.566329 6572 fs_manager.cc:359] Metadata directory not provided
I20260929 19:01:14.566395 6572 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:01:14.566551 6572 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:01:14.567525 6572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/minicluster-data/ts-0/data/instance:
uuid: "137b431b75d748d2bbc3786ad9df1faa"
format_stamp: "Formatted at 2026-09-29 19:01:14 on dist-test-slave-lqkv"
server_key: "a73faf3300b87fcc63f35f825fa3e33b"
server_key_iv: "a5eab158b4fa48e75fc1cc5637052877"
server_key_version: "encryptionkey@0"
I20260929 19:01:14.567864 6572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/minicluster-data/ts-0/wal/instance:
uuid: "137b431b75d748d2bbc3786ad9df1faa"
format_stamp: "Formatted at 2026-09-29 19:01:14 on dist-test-slave-lqkv"
server_key: "a73faf3300b87fcc63f35f825fa3e33b"
server_key_iv: "a5eab158b4fa48e75fc1cc5637052877"
server_key_version: "encryptionkey@0"
I20260929 19:01:14.569191 6572 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260929 19:01:14.569868 6587 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:14.570025 6572 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 19:01:14.570132 6572 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/minicluster-data/ts-0/wal
uuid: "137b431b75d748d2bbc3786ad9df1faa"
format_stamp: "Formatted at 2026-09-29 19:01:14 on dist-test-slave-lqkv"
server_key: "a73faf3300b87fcc63f35f825fa3e33b"
server_key_iv: "a5eab158b4fa48e75fc1cc5637052877"
server_key_version: "encryptionkey@0"
I20260929 19:01:14.570201 6572 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:01:14.581919 6572 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:01:14.583503 6572 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:01:14.583657 6572 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:01:14.583858 6572 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:01:14.584156 6572 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:01:14.584218 6572 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:14.584249 6572 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:01:14.584293 6572 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:14.589870 6572 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46675
I20260929 19:01:14.589890 6700 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46675 every 8 connection(s)
I20260929 19:01:14.590204 6572 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/minicluster-data/ts-0/data/info.pb
Sep 29 19:01:14 dist-test-slave-lqkv krb5kdc[6465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708474, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260929 19:01:14.598224 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUbnLHE/build/release/bin/kudu as pid 6572
I20260929 19:01:14.598344 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/minicluster-data/ts-0/wal/instance
I20260929 19:01:14.598474 6550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:14.590906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33707 (local address 127.0.98.62:42245)
0929 19:01:14.590972 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:14.590974 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:14.591150 (+ 176us) server_negotiation.cc:407] Connection header received
0929 19:01:14.591562 (+ 412us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:14.591562 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:14.591575 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:14.591624 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:01:14.592453 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:14.593015 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:14.593947 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:14.594133 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:14.594187 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:14.596089 (+ 1902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:01:14.596090 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:01:14.596090 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:01:14.596096 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:01:14.597355 (+ 1259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:14.597683 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:14.597684 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:14.597684 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:14.597730 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:14.597920 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:14.597921 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:14.597921 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:14.598010 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:01:14.598056 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0929 19:01:14.598303 (+ 247us) server_negotiation.cc:299] Negotiation successful
0929 19:01:14.598387 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260929 19:01:14.598563 392 external_mini_cluster.cc:1428] Setting key 8d1585192a9255e649d975a87589c911
I20260929 19:01:14.598636 6703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:14.590879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42245 (local address 127.0.98.1:33707)
0929 19:01:14.591032 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:14.591063 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0929 19:01:14.591485 (+ 422us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:01:14.591673 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:01:14.591676 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:01:14.592199 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:01:14.592362 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:14.592365 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:14.593108 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:14.593108 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:14.593875 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:14.593877 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:14.594207 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:14.594207 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:14.594294 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:14.594753 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:01:14.594760 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:01:14.596008 (+ 1248us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:01:14.597422 (+ 1414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:01:14.597427 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:01:14.597428 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:01:14.597622 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:01:14.597784 (+ 162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:01:14.597784 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:01:14.597785 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:01:14.597870 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:01:14.598061 (+ 191us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:01:14.598064 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:01:14.598166 (+ 102us) client_negotiation.cc:769] Sending connection context
0929 19:01:14.598275 (+ 109us) client_negotiation.cc:240] Negotiation successful
0929 19:01:14.598416 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":88,"threads_started":1}
I20260929 19:01:14.599110 6701 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42245
I20260929 19:01:14.599203 6701 heartbeater.cc:464] Registering TS with master...
I20260929 19:01:14.599386 6701 heartbeater.cc:511] Master 127.0.98.62:42245 requested a full tablet report, sending...
I20260929 19:01:14.599928 6510 ts_manager.cc:194] Registered new tserver with Master: 137b431b75d748d2bbc3786ad9df1faa (127.0.98.1:46675)
I20260929 19:01:14.600757 6510 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:33707
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260929 19:01:14.621574 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUbnLHE/build/release/bin/kudu
/tmp/dist-test-taskUbnLHE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42245
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:01:14.699005 6706 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:01:14.699188 6706 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:01:14.699210 6706 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:01:14.700312 6706 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:01:14.700351 6706 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:01:14.700377 6706 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 19:01:14.701782 6706 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42245
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6706
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 18:09:13 UTC on ccycloud.quasar-hpkalf.root.comops.site
build id 4680913
I20260929 19:01:14.702018 6706 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:01:14.702231 6706 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:01:14.707103 6711 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:01:14.707079 6714 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:01:14.707120 6712 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:01:14.707193 6706 server_base.cc:1034] running on GCE node
I20260929 19:01:14.707472 6706 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:01:14.707741 6706 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:01:14.708859 6706 hybrid_clock.cc:648] HybridClock initialized: now 1790708474708834 us; error 34 us; skew 500 ppm
Sep 29 19:01:14 dist-test-slave-lqkv krb5kdc[6465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708474, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:01:14.711117 6706 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260929 19:01:14.711557 6706 webserver.cc:460] Webserver started at http://127.0.98.2:46503/ using document root <none> and password file <none>
I20260929 19:01:14.711781 6706 fs_manager.cc:359] Metadata directory not provided
I20260929 19:01:14.711846 6706 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:01:14.711973 6706 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:01:14.713008 6706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/minicluster-data/ts-1/data/instance:
uuid: "84528791e3a442dda71a633e7a3b6140"
format_stamp: "Formatted at 2026-09-29 19:01:14 on dist-test-slave-lqkv"
server_key: "420b359ac04448c53dcbe214dcb7df5c"
server_key_iv: "b124fdffbcfb3b4ed59faa37ab6df22a"
server_key_version: "encryptionkey@0"
I20260929 19:01:14.713353 6706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/minicluster-data/ts-1/wal/instance:
uuid: "84528791e3a442dda71a633e7a3b6140"
format_stamp: "Formatted at 2026-09-29 19:01:14 on dist-test-slave-lqkv"
server_key: "420b359ac04448c53dcbe214dcb7df5c"
server_key_iv: "b124fdffbcfb3b4ed59faa37ab6df22a"
server_key_version: "encryptionkey@0"
I20260929 19:01:14.714589 6706 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 19:01:14.715317 6721 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:14.715482 6706 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 19:01:14.715576 6706 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/minicluster-data/ts-1/wal
uuid: "84528791e3a442dda71a633e7a3b6140"
format_stamp: "Formatted at 2026-09-29 19:01:14 on dist-test-slave-lqkv"
server_key: "420b359ac04448c53dcbe214dcb7df5c"
server_key_iv: "b124fdffbcfb3b4ed59faa37ab6df22a"
server_key_version: "encryptionkey@0"
I20260929 19:01:14.715655 6706 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:01:14.743570 6706 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:01:14.745369 6706 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:01:14.745527 6706 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:01:14.745744 6706 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:01:14.746035 6706 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:01:14.746086 6706 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:14.746119 6706 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:01:14.746146 6706 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:14.751734 6706 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35351
I20260929 19:01:14.751785 6834 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35351 every 8 connection(s)
I20260929 19:01:14.752058 6706 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/minicluster-data/ts-1/data/info.pb
I20260929 19:01:14.755627 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUbnLHE/build/release/bin/kudu as pid 6706
I20260929 19:01:14.755764 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/minicluster-data/ts-1/wal/instance
I20260929 19:01:14.755980 392 external_mini_cluster.cc:1428] Setting key 68211fb0ea6e62ef17e1c83ef69df576
Sep 29 19:01:14 dist-test-slave-lqkv krb5kdc[6465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708474, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260929 19:01:14.761718 6837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:14.752664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42245 (local address 127.0.98.2:55225)
0929 19:01:14.752824 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:14.752869 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 19:01:14.753377 (+ 508us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:01:14.753606 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:01:14.753609 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:01:14.754139 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:01:14.754318 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:14.754322 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:14.754927 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:14.754928 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:14.755918 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:14.755920 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:14.756383 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:14.756384 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:14.756484 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:14.757030 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:01:14.757037 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:01:14.758525 (+ 1488us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:01:14.760354 (+ 1829us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:01:14.760361 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:01:14.760363 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:01:14.760588 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:01:14.760868 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:01:14.760873 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:01:14.760873 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:01:14.760986 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:01:14.761288 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:01:14.761291 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:01:14.761395 (+ 104us) client_negotiation.cc:769] Sending connection context
0929 19:01:14.761466 (+ 71us) client_negotiation.cc:240] Negotiation successful
0929 19:01:14.761572 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":78,"threads_started":1}
I20260929 19:01:14.761765 6550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:14.752712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55225 (local address 127.0.98.62:42245)
0929 19:01:14.752755 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:14.752757 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:14.752959 (+ 202us) server_negotiation.cc:407] Connection header received
0929 19:01:14.753495 (+ 536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:14.753496 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:14.753506 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:14.753555 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:01:14.754388 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:14.754857 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:14.755995 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:14.756296 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:14.756365 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:14.758618 (+ 2253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:01:14.758619 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:01:14.758619 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:01:14.758627 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:01:14.760256 (+ 1629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:14.760674 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:14.760675 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:14.760676 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:14.760743 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:14.761056 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:14.761057 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:14.761058 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:14.761201 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:01:14.761297 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0929 19:01:14.761533 (+ 236us) server_negotiation.cc:299] Negotiation successful
0929 19:01:14.761618 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260929 19:01:14.762271 6835 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42245
I20260929 19:01:14.762382 6835 heartbeater.cc:464] Registering TS with master...
I20260929 19:01:14.762547 6835 heartbeater.cc:511] Master 127.0.98.62:42245 requested a full tablet report, sending...
I20260929 19:01:14.762880 6510 ts_manager.cc:194] Registered new tserver with Master: 84528791e3a442dda71a633e7a3b6140 (127.0.98.2:35351)
I20260929 19:01:14.763357 6510 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:55225
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260929 19:01:14.779423 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUbnLHE/build/release/bin/kudu
/tmp/dist-test-taskUbnLHE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42245
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:01:14.857129 6840 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:01:14.857329 6840 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:01:14.857363 6840 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:01:14.858454 6840 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:01:14.858517 6840 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:01:14.858560 6840 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 19:01:14.859972 6840 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42245
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6840
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 18:09:13 UTC on ccycloud.quasar-hpkalf.root.comops.site
build id 4680913
I20260929 19:01:14.860235 6840 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:01:14.860451 6840 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:01:14.865212 6845 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:01:14.865237 6846 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:01:14.865338 6840 server_base.cc:1034] running on GCE node
W20260929 19:01:14.865219 6848 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:01:14.865612 6840 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:01:14.865794 6840 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:01:14.866971 6840 hybrid_clock.cc:648] HybridClock initialized: now 1790708474866956 us; error 43 us; skew 500 ppm
Sep 29 19:01:14 dist-test-slave-lqkv krb5kdc[6465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708474, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:01:14.869483 6840 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260929 19:01:14.869989 6840 webserver.cc:460] Webserver started at http://127.0.98.3:33635/ using document root <none> and password file <none>
I20260929 19:01:14.870234 6840 fs_manager.cc:359] Metadata directory not provided
I20260929 19:01:14.870277 6840 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:01:14.870415 6840 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:01:14.871351 6840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/minicluster-data/ts-2/data/instance:
uuid: "07b2d889f6074dcf998c9ef656c568ca"
format_stamp: "Formatted at 2026-09-29 19:01:14 on dist-test-slave-lqkv"
server_key: "7892eb957b9eb97562bf017f9e6b3843"
server_key_iv: "6eff6dbc37ea0201a8b50bdc01fcbfc6"
server_key_version: "encryptionkey@0"
I20260929 19:01:14.871676 6840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/minicluster-data/ts-2/wal/instance:
uuid: "07b2d889f6074dcf998c9ef656c568ca"
format_stamp: "Formatted at 2026-09-29 19:01:14 on dist-test-slave-lqkv"
server_key: "7892eb957b9eb97562bf017f9e6b3843"
server_key_iv: "6eff6dbc37ea0201a8b50bdc01fcbfc6"
server_key_version: "encryptionkey@0"
I20260929 19:01:14.872931 6840 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 19:01:14.873667 6855 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:14.873847 6840 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 19:01:14.873929 6840 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/minicluster-data/ts-2/wal
uuid: "07b2d889f6074dcf998c9ef656c568ca"
format_stamp: "Formatted at 2026-09-29 19:01:14 on dist-test-slave-lqkv"
server_key: "7892eb957b9eb97562bf017f9e6b3843"
server_key_iv: "6eff6dbc37ea0201a8b50bdc01fcbfc6"
server_key_version: "encryptionkey@0"
I20260929 19:01:14.874013 6840 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:01:14.886627 6840 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:01:14.888273 6840 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:01:14.888417 6840 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:01:14.888629 6840 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:01:14.888967 6840 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:01:14.889017 6840 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:14.889055 6840 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:01:14.889081 6840 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:14.894438 6840 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42589
I20260929 19:01:14.894469 6968 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42589 every 8 connection(s)
I20260929 19:01:14.894748 6840 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/minicluster-data/ts-2/data/info.pb
Sep 29 19:01:14 dist-test-slave-lqkv krb5kdc[6465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708474, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260929 19:01:14.902635 6550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:14.895393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58481 (local address 127.0.98.62:42245)
0929 19:01:14.895462 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:14.895464 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:14.895583 (+ 119us) server_negotiation.cc:407] Connection header received
0929 19:01:14.895974 (+ 391us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:14.895975 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:14.895985 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:14.896034 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:01:14.896907 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:14.897380 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:14.898265 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:14.898430 (+ 165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:14.898478 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:14.900383 (+ 1905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:01:14.900384 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:01:14.900384 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:01:14.900390 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:01:14.901548 (+ 1158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:14.901879 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:14.901880 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:14.901880 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:14.901925 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:14.902112 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:14.902113 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:14.902113 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:14.902200 (+ 87us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:01:14.902248 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0929 19:01:14.902525 (+ 277us) server_negotiation.cc:299] Negotiation successful
0929 19:01:14.902578 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 19:01:14.902832 6971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:14.895351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42245 (local address 127.0.98.3:58481)
0929 19:01:14.895483 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:14.895513 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0929 19:01:14.895901 (+ 388us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:01:14.896079 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:01:14.896081 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:01:14.896574 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:01:14.896759 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:14.896762 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:14.897452 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:14.897452 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:14.898199 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:14.898200 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:14.898494 (+ 294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:14.898495 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:14.898584 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:14.899051 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:01:14.899058 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:01:14.900309 (+ 1251us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:01:14.901613 (+ 1304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:01:14.901618 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:01:14.901619 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:01:14.901818 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:01:14.901978 (+ 160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:01:14.901979 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:01:14.901979 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:01:14.902063 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:01:14.902293 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:01:14.902299 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:01:14.902404 (+ 105us) client_negotiation.cc:769] Sending connection context
0929 19:01:14.902494 (+ 90us) client_negotiation.cc:240] Negotiation successful
0929 19:01:14.902599 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":82,"threads_started":1}
I20260929 19:01:14.903311 6969 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42245
I20260929 19:01:14.903439 6969 heartbeater.cc:464] Registering TS with master...
I20260929 19:01:14.903535 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUbnLHE/build/release/bin/kudu as pid 6840
I20260929 19:01:14.903633 6969 heartbeater.cc:511] Master 127.0.98.62:42245 requested a full tablet report, sending...
I20260929 19:01:14.903644 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/minicluster-data/ts-2/wal/instance
I20260929 19:01:14.903864 392 external_mini_cluster.cc:1428] Setting key 52b8c1bf51b4935f48952b55b4411269
I20260929 19:01:14.904018 6510 ts_manager.cc:194] Registered new tserver with Master: 07b2d889f6074dcf998c9ef656c568ca (127.0.98.3:42589)
I20260929 19:01:14.904394 6510 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:58481
I20260929 19:01:14.904682 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/krb5kdc/oryx_127.0.98.61.keytab.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/krb5kdc/oryx_127.0.98.61.keytab.
I20260929 19:01:14.926952 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUbnLHE/build/release/bin/kudu
/tmp/dist-test-taskUbnLHE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:42823
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42245,127.0.98.61:42823
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:01:15.003476 6974 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:01:15.003687 6974 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:01:15.003723 6974 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:01:15.005038 6974 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 19:01:15.005092 6974 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:01:15.005116 6974 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:01:15.005146 6974 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 19:01:15.005167 6974 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 19:01:15.006661 6974 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42245,127.0.98.61:42823
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:42823
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6974
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 18:09:13 UTC on ccycloud.quasar-hpkalf.root.comops.site
build id 4680913
W20260929 19:01:15.006788 6974 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.98.62:42245,127.0.98.61:42823'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260929 19:01:15.006947 6974 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:01:15.007154 6974 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:01:15.011669 6982 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:01:15.011799 6974 server_base.cc:1034] running on GCE node
W20260929 19:01:15.011695 6979 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:01:15.011695 6980 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:01:15.012094 6974 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:01:15.012374 6974 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:01:15.013547 6974 hybrid_clock.cc:648] HybridClock initialized: now 1790708475013533 us; error 23 us; skew 500 ppm
Sep 29 19:01:15 dist-test-slave-lqkv krb5kdc[6465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708475, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:01:15.015753 6974 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260929 19:01:15.016178 6974 webserver.cc:460] Webserver started at http://127.0.98.61:33747/ using document root <none> and password file <none>
I20260929 19:01:15.016363 6974 fs_manager.cc:359] Metadata directory not provided
I20260929 19:01:15.016402 6974 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:01:15.016486 6974 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:01:15.017463 6974 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/minicluster-data/master-1/data/instance:
uuid: "f5eac47223df4bf997b0f24a92898e8f"
format_stamp: "Formatted at 2026-09-29 19:01:15 on dist-test-slave-lqkv"
server_key: "b2c4b7c7e736d5c30f0ad3011d8fc5b4"
server_key_iv: "c034776b3f9d06fe520653de51439c0a"
server_key_version: "encryptionkey@0"
I20260929 19:01:15.017740 6974 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/minicluster-data/master-1/wal/instance:
uuid: "f5eac47223df4bf997b0f24a92898e8f"
format_stamp: "Formatted at 2026-09-29 19:01:15 on dist-test-slave-lqkv"
server_key: "b2c4b7c7e736d5c30f0ad3011d8fc5b4"
server_key_iv: "c034776b3f9d06fe520653de51439c0a"
server_key_version: "encryptionkey@0"
I20260929 19:01:15.018955 6974 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260929 19:01:15.019595 6989 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:15.019764 6974 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 19:01:15.019836 6974 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/minicluster-data/master-1/wal
uuid: "f5eac47223df4bf997b0f24a92898e8f"
format_stamp: "Formatted at 2026-09-29 19:01:15 on dist-test-slave-lqkv"
server_key: "b2c4b7c7e736d5c30f0ad3011d8fc5b4"
server_key_iv: "c034776b3f9d06fe520653de51439c0a"
server_key_version: "encryptionkey@0"
I20260929 19:01:15.019881 6974 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:01:15.033214 6974 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:01:15.035269 6974 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:01:15.035378 6974 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:01:15.039093 6974 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:42823
I20260929 19:01:15.039153 7041 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:42823 every 8 connection(s)
I20260929 19:01:15.039446 6974 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/minicluster-data/master-1/data/info.pb
I20260929 19:01:15.040477 7042 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:01:15.040769 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUbnLHE/build/release/bin/kudu as pid 6974
I20260929 19:01:15.042163 7042 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42245 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 19:01:15 dist-test-slave-lqkv krb5kdc[6465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708475, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 29 19:01:15 dist-test-slave-lqkv krb5kdc[6465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708474, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260929 19:01:15.050640 6550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:15.042788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54560 (local address 127.0.98.62:42245)
0929 19:01:15.042848 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:15.042851 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:15.043001 (+ 150us) server_negotiation.cc:407] Connection header received
0929 19:01:15.043070 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:15.043071 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:15.043081 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:15.043138 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:01:15.043583 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:15.044058 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:15.044956 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:15.045146 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:15.045225 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:15.047286 (+ 2061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:01:15.047287 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:01:15.047287 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:01:15.047293 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:01:15.048956 (+ 1663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:15.049426 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:15.049427 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:15.049428 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:15.049484 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:15.050006 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:15.050007 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:15.050007 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:15.050213 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:01:15.050269 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0929 19:01:15.050475 (+ 206us) server_negotiation.cc:299] Negotiation successful
0929 19:01:15.050552 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260929 19:01:15.050803 7045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:15.041471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44926 (local address 127.0.98.61:42823)
0929 19:01:15.041650 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:15.041659 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:15.041688 (+ 29us) server_negotiation.cc:407] Connection header received
0929 19:01:15.042082 (+ 394us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:15.042088 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:15.042131 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:15.042271 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:01:15.042999 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:15.043822 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:15.044547 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:15.044998 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:15.045105 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:15.047766 (+ 2661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:01:15.047773 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:01:15.047774 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:01:15.047781 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:01:15.049484 (+ 1703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:15.049808 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:15.049811 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:15.049811 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:15.049915 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:15.050145 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:15.050146 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:15.050146 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:15.050364 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:01:15.050514 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0929 19:01:15.050604 (+ 90us) server_negotiation.cc:299] Negotiation successful
0929 19:01:15.050660 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":133,"thread_start_us":86,"threads_started":1}
I20260929 19:01:15.050803 7046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:15.042694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42245 (local address 127.0.0.1:54560)
0929 19:01:15.042885 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:15.042923 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 19:01:15.043010 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:01:15.043156 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:01:15.043158 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:01:15.043406 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:01:15.043488 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:15.043493 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:15.044161 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:15.044162 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:15.044859 (+ 697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:15.044861 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:15.045233 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:15.045235 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:15.045313 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:15.045787 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:01:15.045795 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:01:15.047205 (+ 1410us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:01:15.049026 (+ 1821us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:01:15.049030 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:01:15.049032 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:01:15.049351 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:01:15.049825 (+ 474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:01:15.049825 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:01:15.049826 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:01:15.049943 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:01:15.050281 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:01:15.050285 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:01:15.050358 (+ 73us) client_negotiation.cc:769] Sending connection context
0929 19:01:15.050446 (+ 88us) client_negotiation.cc:240] Negotiation successful
0929 19:01:15.050588 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"mutex_wait_us":21,"thread_start_us":117,"threads_started":1}
I20260929 19:01:15.051374 7042 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42823 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 19:01:15 dist-test-slave-lqkv krb5kdc[6465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708475, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260929 19:01:15.057080 7046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:15.051598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42823 (local address 127.0.0.1:44932)
0929 19:01:15.051658 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:15.051668 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 19:01:15.051713 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:01:15.051864 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:01:15.051865 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:01:15.051973 (+ 108us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:01:15.052014 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:15.052015 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:15.052558 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:15.052559 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:15.053081 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:15.053083 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:15.053371 (+ 288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:15.053371 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:15.053419 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:15.053830 (+ 411us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:01:15.053834 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:01:15.054929 (+ 1095us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:01:15.056233 (+ 1304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:01:15.056233 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:01:15.056234 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:01:15.056403 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:01:15.056574 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:01:15.056574 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:01:15.056574 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:01:15.056621 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:01:15.056865 (+ 244us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:01:15.056866 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:01:15.056903 (+ 37us) client_negotiation.cc:769] Sending connection context
0929 19:01:15.056956 (+ 53us) client_negotiation.cc:240] Negotiation successful
0929 19:01:15.057024 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260929 19:01:15.057138 7045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:15.051628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44932 (local address 127.0.98.61:42823)
0929 19:01:15.051658 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:15.051659 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:15.051759 (+ 100us) server_negotiation.cc:407] Connection header received
0929 19:01:15.051773 (+ 14us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:15.051773 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:15.051779 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:15.051824 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:01:15.052082 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:15.052500 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:15.053141 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:15.053313 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:15.053360 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:15.054998 (+ 1638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:01:15.054999 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:01:15.054999 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:01:15.055005 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:01:15.056148 (+ 1143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:15.056475 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:15.056476 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:15.056476 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:15.056520 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:15.056671 (+ 151us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:15.056672 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:15.056672 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:15.056759 (+ 87us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:01:15.056845 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 19:01:15.056986 (+ 141us) server_negotiation.cc:299] Negotiation successful
0929 19:01:15.057067 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":25,"server-negotiator.queue_time_us":13}
I20260929 19:01:15.058238 7042 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5eac47223df4bf997b0f24a92898e8f: Bootstrap starting.
I20260929 19:01:15.058990 7042 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f5eac47223df4bf997b0f24a92898e8f: Neither blocks nor log segments found. Creating new log.
I20260929 19:01:15.059258 7042 log.cc:824] T 00000000000000000000000000000000 P f5eac47223df4bf997b0f24a92898e8f: Log is configured to *not* fsync() on all Append() calls
I20260929 19:01:15.059901 7042 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5eac47223df4bf997b0f24a92898e8f: No bootstrap required, opened a new log
I20260929 19:01:15.061265 7042 raft_consensus.cc:348] T 00000000000000000000000000000000 P f5eac47223df4bf997b0f24a92898e8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea566150fd024ed9b9485efbb8ee914c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42245 } } peers { permanent_uuid: "f5eac47223df4bf997b0f24a92898e8f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42823 } }
I20260929 19:01:15.061400 7042 raft_consensus.cc:374] T 00000000000000000000000000000000 P f5eac47223df4bf997b0f24a92898e8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:01:15.061443 7042 raft_consensus.cc:729] T 00000000000000000000000000000000 P f5eac47223df4bf997b0f24a92898e8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5eac47223df4bf997b0f24a92898e8f, State: Initialized, Role: FOLLOWER
I20260929 19:01:15.061561 7042 consensus_queue.cc:260] T 00000000000000000000000000000000 P f5eac47223df4bf997b0f24a92898e8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea566150fd024ed9b9485efbb8ee914c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42245 } } peers { permanent_uuid: "f5eac47223df4bf997b0f24a92898e8f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42823 } }
I20260929 19:01:15.061767 7048 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5eac47223df4bf997b0f24a92898e8f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea566150fd024ed9b9485efbb8ee914c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42245 } } peers { permanent_uuid: "f5eac47223df4bf997b0f24a92898e8f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42823 } } }
I20260929 19:01:15.061838 7042 sys_catalog.cc:564] T 00000000000000000000000000000000 P f5eac47223df4bf997b0f24a92898e8f [sys.catalog]: configured and running, proceeding with master startup.
I20260929 19:01:15.061838 7048 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5eac47223df4bf997b0f24a92898e8f [sys.catalog]: This master's current role is: FOLLOWER
W20260929 19:01:15.063360 7059 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f5eac47223df4bf997b0f24a92898e8f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 19:01:15.063412 7059 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 19:01:15.064743 6974 master_runner.cc:428] Detected that this master f5eac47223df4bf997b0f24a92898e8f is joining an existing cluster
I20260929 19:01:15.064793 6974 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.61:42823
I20260929 19:01:15.072306 6550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:15.065748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54570 (local address 127.0.98.62:42245)
0929 19:01:15.065796 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:15.065798 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:15.065867 (+ 69us) server_negotiation.cc:407] Connection header received
0929 19:01:15.065965 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:15.065966 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:15.065977 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:15.066009 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:01:15.066496 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:15.067255 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:15.068112 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:15.068327 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:15.068411 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:15.069403 (+ 992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:01:15.069404 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:01:15.069404 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:01:15.069411 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:01:15.071004 (+ 1593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:15.071461 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:15.071462 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:15.071463 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:15.071515 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:15.071727 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:15.071728 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:15.071728 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:15.071847 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:01:15.071905 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0929 19:01:15.072093 (+ 188us) server_negotiation.cc:299] Negotiation successful
0929 19:01:15.072183 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260929 19:01:15.072305 7065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:15.065688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42245 (local address 127.0.0.1:54570)
0929 19:01:15.065806 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:15.065816 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 19:01:15.065891 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:01:15.066069 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:01:15.066070 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:01:15.066302 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:01:15.066405 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:15.066407 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:15.067341 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:15.067342 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:15.068026 (+ 684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:15.068027 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:15.068414 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:15.068414 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:15.068476 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:15.068893 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:01:15.068898 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:01:15.069320 (+ 422us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:01:15.071105 (+ 1785us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:01:15.071106 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:01:15.071106 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:01:15.071383 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:01:15.071591 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:01:15.071592 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:01:15.071593 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:01:15.071666 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:01:15.071927 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:01:15.071928 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:01:15.071970 (+ 42us) client_negotiation.cc:769] Sending connection context
0929 19:01:15.072062 (+ 92us) client_negotiation.cc:240] Negotiation successful
0929 19:01:15.072172 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"mutex_wait_us":71,"thread_start_us":69,"threads_started":1}
I20260929 19:01:15.072611 7045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:15.065893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44944 (local address 127.0.98.61:42823)
0929 19:01:15.065923 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:15.065924 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:15.066030 (+ 106us) server_negotiation.cc:407] Connection header received
0929 19:01:15.066089 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:15.066090 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:15.066098 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:15.066150 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:01:15.066875 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:15.067327 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:15.068417 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:15.068690 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:15.068726 (+ 36us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:15.069721 (+ 995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:01:15.069723 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:01:15.069723 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:01:15.069729 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:01:15.071409 (+ 1680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:15.071774 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:15.071781 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:15.071782 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:15.071821 (+ 39us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:15.072045 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:15.072046 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:15.072046 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:15.072176 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:01:15.072232 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0929 19:01:15.072427 (+ 195us) server_negotiation.cc:299] Negotiation successful
0929 19:01:15.072497 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260929 19:01:15.072700 7066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:15.065860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42823 (local address 127.0.0.1:44944)
0929 19:01:15.065978 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:15.065987 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 19:01:15.066035 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:01:15.066151 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:01:15.066151 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:01:15.066681 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:01:15.066718 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:15.066719 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:15.067396 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:15.067397 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:15.068325 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:15.068327 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:15.068789 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:15.068790 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:15.068893 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:15.069295 (+ 402us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:01:15.069299 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:01:15.069632 (+ 333us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:01:15.071493 (+ 1861us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:01:15.071494 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:01:15.071494 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:01:15.071697 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:01:15.071881 (+ 184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:01:15.071882 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:01:15.071889 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:01:15.071948 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:01:15.072252 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:01:15.072253 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:01:15.072294 (+ 41us) client_negotiation.cc:769] Sending connection context
0929 19:01:15.072389 (+ 95us) client_negotiation.cc:240] Negotiation successful
0929 19:01:15.072548 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"mutex_wait_us":73,"thread_start_us":85,"threads_started":1}
Sep 29 19:01:15 dist-test-slave-lqkv krb5kdc[6465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708474, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260929 19:01:15.082821 7067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:15.076017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42823 (local address 127.0.0.1:44948)
0929 19:01:15.076136 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:15.076175 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 19:01:15.076242 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:01:15.076384 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:01:15.076386 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:01:15.076616 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:01:15.076653 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:15.076655 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:15.077522 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:15.077523 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:15.078066 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:15.078067 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:15.078508 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:15.078509 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:15.078567 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:15.078990 (+ 423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:01:15.078994 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:01:15.080406 (+ 1412us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:01:15.081939 (+ 1533us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:01:15.081944 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:01:15.081946 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:01:15.082108 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:01:15.082312 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:01:15.082312 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:01:15.082313 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:01:15.082359 (+ 46us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:01:15.082591 (+ 232us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:01:15.082595 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:01:15.082631 (+ 36us) client_negotiation.cc:769] Sending connection context
0929 19:01:15.082689 (+ 58us) client_negotiation.cc:240] Negotiation successful
0929 19:01:15.082762 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":61,"threads_started":1}
I20260929 19:01:15.082878 7045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:15.076087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44948 (local address 127.0.98.61:42823)
0929 19:01:15.076123 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:15.076125 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:15.076245 (+ 120us) server_negotiation.cc:407] Connection header received
0929 19:01:15.076295 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:15.076296 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:15.076305 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:15.076377 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:01:15.076726 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:15.077438 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:15.078138 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:15.078437 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:15.078509 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:15.080485 (+ 1976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:01:15.080486 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:01:15.080486 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:01:15.080492 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:01:15.081864 (+ 1372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:15.082193 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:15.082194 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:15.082194 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:15.082248 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:15.082421 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:15.082422 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:15.082422 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:15.082533 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:01:15.082594 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0929 19:01:15.082732 (+ 138us) server_negotiation.cc:299] Negotiation successful
0929 19:01:15.082799 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260929 19:01:15.083690 6509 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.61:42823: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "f5eac47223df4bf997b0f24a92898e8f"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.61"
port: 42823
}
attrs {
promote: true
}
}
dest_uuid: "ea566150fd024ed9b9485efbb8ee914c"
cas_config_opid_index: -1
I20260929 19:01:15.083972 6509 consensus_queue.cc:237] T 00000000000000000000000000000000 P ea566150fd024ed9b9485efbb8ee914c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ea566150fd024ed9b9485efbb8ee914c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42245 } } peers { permanent_uuid: "f5eac47223df4bf997b0f24a92898e8f" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 42823 } attrs { promote: true } }
I20260929 19:01:15.084424 7069 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ea566150fd024ed9b9485efbb8ee914c [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER f5eac47223df4bf997b0f24a92898e8f (127.0.98.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ea566150fd024ed9b9485efbb8ee914c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42245 } } peers { permanent_uuid: "f5eac47223df4bf997b0f24a92898e8f" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 42823 } attrs { promote: true } } }
I20260929 19:01:15.084684 7016 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f5eac47223df4bf997b0f24a92898e8f [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:01:15.085353 7069 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.61:42823
I20260929 19:01:15.085459 7068 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea566150fd024ed9b9485efbb8ee914c [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "ea566150fd024ed9b9485efbb8ee914c" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ea566150fd024ed9b9485efbb8ee914c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42245 } } peers { permanent_uuid: "f5eac47223df4bf997b0f24a92898e8f" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 42823 } attrs { promote: true } } }
I20260929 19:01:15.085546 7068 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea566150fd024ed9b9485efbb8ee914c [sys.catalog]: This master's current role is: LEADER
I20260929 19:01:15.085572 7016 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f5eac47223df4bf997b0f24a92898e8f [term 1 FOLLOWER]: Refusing update from remote peer ea566150fd024ed9b9485efbb8ee914c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260929 19:01:15.085891 7068 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ea566150fd024ed9b9485efbb8ee914c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5eac47223df4bf997b0f24a92898e8f" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 42823 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 19:01:15.085949 6974 master.cc:537] Master@127.0.98.61:42823 shutting down...
W20260929 19:01:15.086556 6494 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260929 19:01:15.086992 6494 consensus_peers.cc:487] T 00000000000000000000000000000000 P ea566150fd024ed9b9485efbb8ee914c -> Peer f5eac47223df4bf997b0f24a92898e8f (127.0.98.61:42823): Couldn't send request to peer f5eac47223df4bf997b0f24a92898e8f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260929 19:01:15.088706 6974 raft_consensus.cc:2229] T 00000000000000000000000000000000 P f5eac47223df4bf997b0f24a92898e8f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 19:01:15.088784 6974 raft_consensus.cc:2258] T 00000000000000000000000000000000 P f5eac47223df4bf997b0f24a92898e8f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 19:01:15.088860 6974 tablet_replica.cc:330] T 00000000000000000000000000000000 P f5eac47223df4bf997b0f24a92898e8f: stopping tablet replica
I20260929 19:01:15.101347 6974 master.cc:552] Master@127.0.98.61:42823 shutdown complete.
I20260929 19:01:15.101395 6974 master_runner.cc:305] Clearing existing system tablet
I20260929 19:01:15.101517 6974 fs_manager.cc:359] Metadata directory not provided
I20260929 19:01:15.101562 6974 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:01:15.102243 6974 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.001s
I20260929 19:01:15.102746 7073 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:15.102893 6974 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:01:15.102957 6974 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:15.102982 6974 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/minicluster-data/master-1/wal
uuid: "f5eac47223df4bf997b0f24a92898e8f"
format_stamp: "Formatted at 2026-09-29 19:01:15 on dist-test-slave-lqkv"
server_key: "b2c4b7c7e736d5c30f0ad3011d8fc5b4"
server_key_iv: "c034776b3f9d06fe520653de51439c0a"
server_key_version: "encryptionkey@0"
I20260929 19:01:15.103670 6974 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P f5eac47223df4bf997b0f24a92898e8f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 19:01:15.105412 6974 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P f5eac47223df4bf997b0f24a92898e8f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260929 19:01:15.105476 6974 log.cc:1196] T 00000000000000000000000000000000 P f5eac47223df4bf997b0f24a92898e8f: Deleting WAL directory at /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260929 19:01:15.105665 6974 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P f5eac47223df4bf997b0f24a92898e8f: Deleting consensus metadata
I20260929 19:01:15.105773 6974 master_runner.cc:315] Copying system tablet from 127.0.98.62:42245
I20260929 19:01:15.106357 6974 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P f5eac47223df4bf997b0f24a92898e8f: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:42245
I20260929 19:01:15.112273 6550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:15.106672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54586 (local address 127.0.98.62:42245)
0929 19:01:15.106716 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:15.106723 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:15.106760 (+ 37us) server_negotiation.cc:407] Connection header received
0929 19:01:15.106825 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:15.106825 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:15.106835 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:15.106893 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:01:15.107145 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:15.107913 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:15.108590 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:15.108922 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:15.108989 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:15.109856 (+ 867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:01:15.109857 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:01:15.109858 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:01:15.109864 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:01:15.111272 (+ 1408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:15.111577 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:15.111579 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:15.111579 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:15.111649 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:15.111809 (+ 160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:15.111810 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:15.111811 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:15.111933 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:01:15.112003 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 19:01:15.112133 (+ 130us) server_negotiation.cc:299] Negotiation successful
0929 19:01:15.112195 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260929 19:01:15.112295 7079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:15.106605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42245 (local address 127.0.0.1:54586)
0929 19:01:15.106703 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:15.106713 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 19:01:15.106768 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:01:15.106896 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:01:15.106897 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:01:15.107031 (+ 134us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:01:15.107080 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:15.107082 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:15.107988 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:15.107989 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:15.108516 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:15.108517 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:15.108987 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:15.108988 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:15.109035 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:15.109485 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:01:15.109489 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:01:15.109776 (+ 287us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:01:15.111350 (+ 1574us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:01:15.111350 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:01:15.111351 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:01:15.111508 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:01:15.111711 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:01:15.111711 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:01:15.111711 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:01:15.111757 (+ 46us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:01:15.111992 (+ 235us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:01:15.111993 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:01:15.112022 (+ 29us) client_negotiation.cc:769] Sending connection context
0929 19:01:15.112105 (+ 83us) client_negotiation.cc:240] Negotiation successful
0929 19:01:15.112194 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80,"thread_start_us":66,"threads_started":1}
I20260929 19:01:15.112677 6530 tablet_copy_service.cc:140] P ea566150fd024ed9b9485efbb8ee914c: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer f5eac47223df4bf997b0f24a92898e8f ({username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:54586)
I20260929 19:01:15.112743 6530 tablet_copy_service.cc:161] P ea566150fd024ed9b9485efbb8ee914c: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer f5eac47223df4bf997b0f24a92898e8f at {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:54586: session id = f5eac47223df4bf997b0f24a92898e8f-00000000000000000000000000000000
I20260929 19:01:15.113300 6530 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P ea566150fd024ed9b9485efbb8ee914c: Tablet Copy: opened 0 blocks and 1 log segments
I20260929 19:01:15.113868 6974 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260929 19:01:15.114885 6974 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P f5eac47223df4bf997b0f24a92898e8f: tablet copy: Starting download of 0 data blocks...
I20260929 19:01:15.115034 6974 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P f5eac47223df4bf997b0f24a92898e8f: tablet copy: Starting download of 1 WAL segments...
I20260929 19:01:15.115803 6974 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P f5eac47223df4bf997b0f24a92898e8f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260929 19:01:15.116585 6530 tablet_copy_service.cc:342] P ea566150fd024ed9b9485efbb8ee914c: Request end of tablet copy session f5eac47223df4bf997b0f24a92898e8f-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:54586
I20260929 19:01:15.116660 6530 tablet_copy_service.cc:433] P ea566150fd024ed9b9485efbb8ee914c: ending tablet copy session f5eac47223df4bf997b0f24a92898e8f-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer f5eac47223df4bf997b0f24a92898e8f
I20260929 19:01:15.119146 6974 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:01:15.120157 7081 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:01:15.120321 7083 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:01:15.120321 7080 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:01:15.120574 6974 server_base.cc:1034] running on GCE node
I20260929 19:01:15.120677 6974 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:01:15.120795 6974 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:01:15.121944 6974 hybrid_clock.cc:648] HybridClock initialized: now 1790708475121936 us; error 21 us; skew 500 ppm
Sep 29 19:01:15 dist-test-slave-lqkv krb5kdc[6465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708475, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:01:15.123103 6974 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260929 19:01:15.123338 6974 webserver.cc:460] Webserver started at http://127.0.98.61:46703/ using document root <none> and password file <none>
I20260929 19:01:15.123476 6974 fs_manager.cc:359] Metadata directory not provided
I20260929 19:01:15.123514 6974 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:01:15.124076 6974 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.001s
I20260929 19:01:15.124547 7090 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:15.124663 6974 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:15.124706 6974 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/minicluster-data/master-1/wal
uuid: "f5eac47223df4bf997b0f24a92898e8f"
format_stamp: "Formatted at 2026-09-29 19:01:15 on dist-test-slave-lqkv"
server_key: "b2c4b7c7e736d5c30f0ad3011d8fc5b4"
server_key_iv: "c034776b3f9d06fe520653de51439c0a"
server_key_version: "encryptionkey@0"
I20260929 19:01:15.124755 6974 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:01:15.129513 6974 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:01:15.131081 6974 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:01:15.134037 6974 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:42823
I20260929 19:01:15.134208 7142 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:42823 every 8 connection(s)
I20260929 19:01:15.134389 6974 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/minicluster-data/master-1/data/info.pb
I20260929 19:01:15.135149 7143 sys_catalog.cc:263] Verifying existing consensus state
I20260929 19:01:15.135588 7143 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5eac47223df4bf997b0f24a92898e8f: Bootstrap starting.
I20260929 19:01:15.138324 7143 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5eac47223df4bf997b0f24a92898e8f: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 19:01:15.138537 7143 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f5eac47223df4bf997b0f24a92898e8f: Bootstrap complete.
I20260929 19:01:15.138720 7143 raft_consensus.cc:348] T 00000000000000000000000000000000 P f5eac47223df4bf997b0f24a92898e8f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ea566150fd024ed9b9485efbb8ee914c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42245 } } peers { permanent_uuid: "f5eac47223df4bf997b0f24a92898e8f" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 42823 } attrs { promote: true } }
I20260929 19:01:15.138800 7143 raft_consensus.cc:729] T 00000000000000000000000000000000 P f5eac47223df4bf997b0f24a92898e8f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f5eac47223df4bf997b0f24a92898e8f, State: Initialized, Role: LEARNER
I20260929 19:01:15.138837 7143 consensus_queue.cc:260] T 00000000000000000000000000000000 P f5eac47223df4bf997b0f24a92898e8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ea566150fd024ed9b9485efbb8ee914c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42245 } } peers { permanent_uuid: "f5eac47223df4bf997b0f24a92898e8f" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 42823 } attrs { promote: true } }
I20260929 19:01:15.139008 7143 sys_catalog.cc:564] T 00000000000000000000000000000000 P f5eac47223df4bf997b0f24a92898e8f [sys.catalog]: configured and running, proceeding with master startup.
I20260929 19:01:15.139005 7146 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5eac47223df4bf997b0f24a92898e8f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ea566150fd024ed9b9485efbb8ee914c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42245 } } peers { permanent_uuid: "f5eac47223df4bf997b0f24a92898e8f" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 42823 } attrs { promote: true } } }
I20260929 19:01:15.139114 7146 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5eac47223df4bf997b0f24a92898e8f [sys.catalog]: This master's current role is: LEARNER
I20260929 19:01:15.140542 7157 catalog_manager.cc:1259] Loaded cluster ID: 273b8d19129f4d25a0e7c884d5c7d851
I20260929 19:01:15.140736 7157 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f5eac47223df4bf997b0f24a92898e8f: loading cluster ID for follower catalog manager: success
I20260929 19:01:15.141544 7157 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f5eac47223df4bf997b0f24a92898e8f: acquiring CA information for follower catalog manager: success
I20260929 19:01:15.141808 7157 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f5eac47223df4bf997b0f24a92898e8f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 19:01:15.159397 7158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:15.152580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44972 (local address 127.0.98.61:42823)
0929 19:01:15.152691 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:15.152693 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:15.152701 (+ 8us) server_negotiation.cc:407] Connection header received
0929 19:01:15.152719 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:15.152719 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:15.152726 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:15.152789 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:01:15.153488 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:15.154099 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:15.155107 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:15.155284 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:15.155332 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:15.156649 (+ 1317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:01:15.156650 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:01:15.156650 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:01:15.156655 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:01:15.158133 (+ 1478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:15.158552 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:15.158553 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:15.158553 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:15.158613 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:15.158854 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:15.158855 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:15.158855 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:15.158949 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:01:15.159030 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 19:01:15.159229 (+ 199us) server_negotiation.cc:299] Negotiation successful
0929 19:01:15.159286 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83,"thread_start_us":66,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.60@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/krb5kdc/oryx_127.0.98.60.keytab.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/krb5kdc/oryx_127.0.98.60.keytab.
I20260929 19:01:15.183089 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUbnLHE/build/release/bin/kudu
/tmp/dist-test-taskUbnLHE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:39419
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42245,127.0.98.61:42823,127.0.98.60:39419
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:01:15.260346 7161 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:01:15.260526 7161 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:01:15.260547 7161 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:01:15.261706 7161 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 19:01:15.261746 7161 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:01:15.261760 7161 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:01:15.261773 7161 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 19:01:15.261821 7161 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 19:01:15.263289 7161 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42245,127.0.98.61:42823,127.0.98.60:39419
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:39419
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7161
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 18:09:13 UTC on ccycloud.quasar-hpkalf.root.comops.site
build id 4680913
I20260929 19:01:15.263513 7161 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:01:15.263741 7161 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:01:15.268277 7169 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:01:15.268353 7161 server_base.cc:1034] running on GCE node
W20260929 19:01:15.268290 7166 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:01:15.268324 7167 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:01:15.268755 7161 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:01:15.269066 7161 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:01:15.270241 7161 hybrid_clock.cc:648] HybridClock initialized: now 1790708475270223 us; error 27 us; skew 500 ppm
Sep 29 19:01:15 dist-test-slave-lqkv krb5kdc[6465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708475, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:01:15.272475 7161 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260929 19:01:15.272955 7161 webserver.cc:460] Webserver started at http://127.0.98.60:42783/ using document root <none> and password file <none>
I20260929 19:01:15.273281 7161 fs_manager.cc:359] Metadata directory not provided
I20260929 19:01:15.273347 7161 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:01:15.273521 7161 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:01:15.274587 7161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/minicluster-data/master-2/data/instance:
uuid: "0908a5f3d25543efb7c1f184cd5178f4"
format_stamp: "Formatted at 2026-09-29 19:01:15 on dist-test-slave-lqkv"
server_key: "a1b90761f6adc10e05f7d41cad3b3c4b"
server_key_iv: "ae7a2d872241e0bc1ac48334e26ec360"
server_key_version: "encryptionkey@0"
I20260929 19:01:15.274940 7161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/minicluster-data/master-2/wal/instance:
uuid: "0908a5f3d25543efb7c1f184cd5178f4"
format_stamp: "Formatted at 2026-09-29 19:01:15 on dist-test-slave-lqkv"
server_key: "a1b90761f6adc10e05f7d41cad3b3c4b"
server_key_iv: "ae7a2d872241e0bc1ac48334e26ec360"
server_key_version: "encryptionkey@0"
I20260929 19:01:15.276269 7161 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 19:01:15.277108 7176 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:15.277312 7161 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 19:01:15.277382 7161 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/minicluster-data/master-2/wal
uuid: "0908a5f3d25543efb7c1f184cd5178f4"
format_stamp: "Formatted at 2026-09-29 19:01:15 on dist-test-slave-lqkv"
server_key: "a1b90761f6adc10e05f7d41cad3b3c4b"
server_key_iv: "ae7a2d872241e0bc1ac48334e26ec360"
server_key_version: "encryptionkey@0"
I20260929 19:01:15.277468 7161 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:01:15.291267 7161 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:01:15.292977 7161 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:01:15.293126 7161 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:01:15.297142 7161 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:39419
I20260929 19:01:15.297168 7228 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:39419 every 8 connection(s)
I20260929 19:01:15.297515 7161 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/minicluster-data/master-2/data/info.pb
I20260929 19:01:15.298460 7229 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:01:15.300308 7229 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42245 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 19:01:15 dist-test-slave-lqkv krb5kdc[6465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708475, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260929 19:01:15.307250 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUbnLHE/build/release/bin/kudu as pid 7161
I20260929 19:01:15.309268 6550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:15.301085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54594 (local address 127.0.98.62:42245)
0929 19:01:15.301151 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:15.301154 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:15.301228 (+ 74us) server_negotiation.cc:407] Connection header received
0929 19:01:15.301707 (+ 479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:15.301708 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:15.301720 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:15.301785 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:01:15.302361 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:15.302885 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:15.303961 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:15.304154 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:15.304214 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:15.306379 (+ 2165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:01:15.306380 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:01:15.306380 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:01:15.306387 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:01:15.307810 (+ 1423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:15.308307 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:15.308308 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:15.308309 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:15.308372 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:15.308620 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:15.308621 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:15.308621 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:15.308735 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:01:15.308806 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 19:01:15.309094 (+ 288us) server_negotiation.cc:299] Negotiation successful
0929 19:01:15.309182 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 19:01:15.309428 7231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:15.300941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42245 (local address 127.0.0.1:54594)
0929 19:01:15.301097 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:15.301162 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0929 19:01:15.301605 (+ 443us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:01:15.301849 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:01:15.301853 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:01:15.302086 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:01:15.302257 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:15.302273 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:15.302981 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:15.302982 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:15.303880 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:15.303883 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:15.304227 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:15.304228 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:15.304321 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:15.304938 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:01:15.304945 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:01:15.306302 (+ 1357us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:01:15.307905 (+ 1603us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:01:15.307910 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:01:15.307912 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:01:15.308226 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:01:15.308437 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:01:15.308438 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:01:15.308438 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:01:15.308547 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:01:15.308826 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:01:15.308832 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:01:15.308959 (+ 127us) client_negotiation.cc:769] Sending connection context
0929 19:01:15.309068 (+ 109us) client_negotiation.cc:240] Negotiation successful
0929 19:01:15.309213 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":89,"threads_started":1}
I20260929 19:01:15.310034 7229 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42823 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 19:01:15 dist-test-slave-lqkv krb5kdc[6465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708474, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Sep 29 19:01:15 dist-test-slave-lqkv krb5kdc[6465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708475, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260929 19:01:15.316267 7232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:15.307778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34190 (local address 127.0.98.60:39419)
0929 19:01:15.307932 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:15.307936 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:15.308020 (+ 84us) server_negotiation.cc:407] Connection header received
0929 19:01:15.308044 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:15.308045 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:15.308057 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:15.308151 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:01:15.308667 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:15.309469 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:15.310444 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:15.310772 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:15.310852 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:15.313137 (+ 2285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:01:15.313144 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:01:15.313144 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:01:15.313150 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:01:15.315020 (+ 1870us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:15.315421 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:15.315423 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:15.315423 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:15.315467 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:15.315683 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:15.315684 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:15.315684 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:15.315818 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:01:15.315999 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0929 19:01:15.316039 (+ 40us) server_negotiation.cc:299] Negotiation successful
0929 19:01:15.316147 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":75,"threads_started":1}
I20260929 19:01:15.317085 7158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:15.310481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44978 (local address 127.0.98.61:42823)
0929 19:01:15.310541 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:15.310543 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:15.310549 (+ 6us) server_negotiation.cc:407] Connection header received
0929 19:01:15.310567 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:15.310567 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:15.310578 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:15.310642 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:01:15.310884 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:15.311522 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:15.312211 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:15.312425 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:15.312514 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:15.314312 (+ 1798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:01:15.314313 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:01:15.314313 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:01:15.314320 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:01:15.315808 (+ 1488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:15.316278 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:15.316280 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:15.316280 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:15.316337 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:15.316563 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:15.316564 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:15.316564 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:15.316702 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:01:15.316746 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0929 19:01:15.316956 (+ 210us) server_negotiation.cc:299] Negotiation successful
0929 19:01:15.317026 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260929 19:01:15.317142 7231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:15.310248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42823 (local address 127.0.0.1:44978)
0929 19:01:15.310293 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:15.310303 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 19:01:15.310335 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:01:15.310643 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:01:15.310644 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:01:15.310768 (+ 124us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:01:15.310806 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:15.310807 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:15.311639 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:15.311640 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:15.312128 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:15.312129 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:15.312515 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:15.312516 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:15.312581 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:15.313055 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:01:15.313060 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:01:15.314172 (+ 1112us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:01:15.315912 (+ 1740us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:01:15.315914 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:01:15.315914 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:01:15.316185 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:01:15.316415 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:01:15.316416 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:01:15.316416 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:01:15.316471 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:01:15.316793 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:01:15.316794 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:01:15.316851 (+ 57us) client_negotiation.cc:769] Sending connection context
0929 19:01:15.316926 (+ 75us) client_negotiation.cc:240] Negotiation successful
0929 19:01:15.317020 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260929 19:01:15.317461 7229 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39419 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 19:01:15 dist-test-slave-lqkv krb5kdc[6465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708475, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260929 19:01:15.323442 7231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:15.317654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39419 (local address 127.0.0.1:34192)
0929 19:01:15.317700 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:15.317710 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 19:01:15.317742 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:01:15.317870 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:01:15.317870 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:01:15.317967 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:01:15.317996 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:15.317997 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:15.318947 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:15.318948 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:15.319433 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:15.319435 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:15.319936 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:15.319936 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:15.319973 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:15.320341 (+ 368us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:01:15.320345 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:01:15.321567 (+ 1222us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:01:15.322793 (+ 1226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:01:15.322793 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:01:15.322794 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:01:15.322953 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:01:15.323094 (+ 141us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:01:15.323095 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:01:15.323095 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:01:15.323133 (+ 38us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:01:15.323268 (+ 135us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:01:15.323268 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:01:15.323293 (+ 25us) client_negotiation.cc:769] Sending connection context
0929 19:01:15.323335 (+ 42us) client_negotiation.cc:240] Negotiation successful
0929 19:01:15.323393 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260929 19:01:15.323500 7232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:15.317702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34192 (local address 127.0.98.60:39419)
0929 19:01:15.317736 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:15.317737 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:15.317746 (+ 9us) server_negotiation.cc:407] Connection header received
0929 19:01:15.317791 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:15.317792 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:15.317802 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:15.317869 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:01:15.318140 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:15.318862 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:15.319517 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:15.319840 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:15.319937 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:15.321639 (+ 1702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:01:15.321640 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:01:15.321640 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:01:15.321646 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:01:15.322728 (+ 1082us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:15.323014 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:15.323015 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:15.323015 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:15.323057 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:15.323164 (+ 107us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:15.323164 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:15.323164 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:15.323236 (+ 72us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:01:15.323274 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0929 19:01:15.323349 (+ 75us) server_negotiation.cc:299] Negotiation successful
0929 19:01:15.323431 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":32,"server-negotiator.queue_time_us":17}
I20260929 19:01:15.324935 7229 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0908a5f3d25543efb7c1f184cd5178f4: Bootstrap starting.
I20260929 19:01:15.325760 7229 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0908a5f3d25543efb7c1f184cd5178f4: Neither blocks nor log segments found. Creating new log.
I20260929 19:01:15.326058 7229 log.cc:824] T 00000000000000000000000000000000 P 0908a5f3d25543efb7c1f184cd5178f4: Log is configured to *not* fsync() on all Append() calls
I20260929 19:01:15.326668 7229 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0908a5f3d25543efb7c1f184cd5178f4: No bootstrap required, opened a new log
I20260929 19:01:15.327998 7229 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0908a5f3d25543efb7c1f184cd5178f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea566150fd024ed9b9485efbb8ee914c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42245 } } peers { permanent_uuid: "f5eac47223df4bf997b0f24a92898e8f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42823 } } peers { permanent_uuid: "0908a5f3d25543efb7c1f184cd5178f4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39419 } }
I20260929 19:01:15.328133 7229 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0908a5f3d25543efb7c1f184cd5178f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:01:15.328181 7229 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0908a5f3d25543efb7c1f184cd5178f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0908a5f3d25543efb7c1f184cd5178f4, State: Initialized, Role: FOLLOWER
I20260929 19:01:15.328310 7229 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0908a5f3d25543efb7c1f184cd5178f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea566150fd024ed9b9485efbb8ee914c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42245 } } peers { permanent_uuid: "f5eac47223df4bf997b0f24a92898e8f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42823 } } peers { permanent_uuid: "0908a5f3d25543efb7c1f184cd5178f4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39419 } }
I20260929 19:01:15.328574 7229 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0908a5f3d25543efb7c1f184cd5178f4 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 19:01:15.328544 7234 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0908a5f3d25543efb7c1f184cd5178f4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea566150fd024ed9b9485efbb8ee914c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42245 } } peers { permanent_uuid: "f5eac47223df4bf997b0f24a92898e8f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42823 } } peers { permanent_uuid: "0908a5f3d25543efb7c1f184cd5178f4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39419 } } }
I20260929 19:01:15.328687 7234 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0908a5f3d25543efb7c1f184cd5178f4 [sys.catalog]: This master's current role is: FOLLOWER
W20260929 19:01:15.330144 7245 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0908a5f3d25543efb7c1f184cd5178f4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 19:01:15.330204 7245 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 19:01:15.332129 7161 master_runner.cc:428] Detected that this master 0908a5f3d25543efb7c1f184cd5178f4 is joining an existing cluster
I20260929 19:01:15.332180 7161 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.60:39419
I20260929 19:01:15.339527 6550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:15.333111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54608 (local address 127.0.98.62:42245)
0929 19:01:15.333148 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:15.333150 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:15.333490 (+ 340us) server_negotiation.cc:407] Connection header received
0929 19:01:15.333578 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:15.333579 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:15.333588 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:15.333642 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:01:15.333908 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:15.334683 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:15.335714 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:15.335984 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:15.336067 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:15.337121 (+ 1054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:01:15.337122 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:01:15.337123 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:01:15.337128 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:01:15.338454 (+ 1326us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:15.338809 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:15.338809 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:15.338810 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:15.338898 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:15.339098 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:15.339098 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:15.339099 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:15.339191 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:01:15.339241 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0929 19:01:15.339383 (+ 142us) server_negotiation.cc:299] Negotiation successful
0929 19:01:15.339431 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260929 19:01:15.339572 7252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:15.333056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42245 (local address 127.0.0.1:54608)
0929 19:01:15.333415 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:15.333425 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 19:01:15.333521 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:01:15.333645 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:01:15.333645 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:01:15.333786 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:01:15.333834 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:15.333836 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:15.334761 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:15.334761 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:15.335624 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:15.335626 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:15.336098 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:15.336099 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:15.336159 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:15.336644 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:01:15.336648 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:01:15.337045 (+ 397us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:01:15.338536 (+ 1491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:01:15.338537 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:01:15.338537 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:01:15.338738 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:01:15.338967 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:01:15.338968 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:01:15.338968 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:01:15.339032 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:01:15.339253 (+ 221us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:01:15.339254 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:01:15.339294 (+ 40us) client_negotiation.cc:769] Sending connection context
0929 19:01:15.339370 (+ 76us) client_negotiation.cc:240] Negotiation successful
0929 19:01:15.339453 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":82,"threads_started":1}
Sep 29 19:01:15 dist-test-slave-lqkv krb5kdc[6465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708474, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260929 19:01:15.361123 7067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:15.342056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39419 (local address 127.0.0.1:34210)
0929 19:01:15.342097 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:15.342106 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 19:01:15.342144 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:01:15.342344 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:01:15.342345 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:01:15.342491 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:01:15.342568 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:15.342570 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:15.343172 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:15.343173 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:15.343964 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:15.343966 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:15.344279 (+ 313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:15.344280 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:15.344344 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:15.344657 (+ 313us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:01:15.344661 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:01:15.345808 (+ 1147us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:01:15.359929 (+ 14121us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:01:15.359930 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:01:15.359930 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:01:15.360174 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:01:15.360400 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:01:15.360401 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:01:15.360401 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:01:15.360458 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:01:15.360780 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:01:15.360781 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:01:15.360852 (+ 71us) client_negotiation.cc:769] Sending connection context
0929 19:01:15.360906 (+ 54us) client_negotiation.cc:240] Negotiation successful
0929 19:01:15.361019 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260929 19:01:15.361235 7232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:15.334019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34194 (local address 127.0.98.60:39419)
0929 19:01:15.334065 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:15.334067 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:15.339742 (+ 5675us) server_negotiation.cc:407] Connection header received
0929 19:01:15.339820 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:15.339821 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:15.339828 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:15.339851 (+ 23us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:01:15.340185 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:15.340986 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:15.341829 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:15.342103 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:15.342175 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:15.358306 (+ 16131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:01:15.358307 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:01:15.358307 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:01:15.358313 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:01:15.360153 (+ 1840us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:15.360521 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:15.360522 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:15.360522 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:15.360563 (+ 41us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:15.360720 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:15.360720 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:15.360721 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:15.360867 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:01:15.360907 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0929 19:01:15.361073 (+ 166us) server_negotiation.cc:299] Negotiation successful
0929 19:01:15.361143 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260929 19:01:15.361459 7252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:15.333459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39419 (local address 127.0.0.1:34194)
0929 19:01:15.339704 (+ 6245us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:15.339713 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 19:01:15.339763 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:01:15.339895 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:01:15.339896 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:01:15.340063 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:01:15.340107 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:15.340109 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:15.341076 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:15.341076 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:15.341757 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:15.341760 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:15.342213 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:15.342214 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:15.342269 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:15.357700 (+ 15431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:01:15.357706 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:01:15.358211 (+ 505us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:01:15.360247 (+ 2036us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:01:15.360247 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:01:15.360247 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:01:15.360430 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:01:15.360621 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:01:15.360621 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:01:15.360621 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:01:15.360663 (+ 42us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:01:15.360945 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:01:15.360946 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:01:15.360978 (+ 32us) client_negotiation.cc:769] Sending connection context
0929 19:01:15.361302 (+ 324us) client_negotiation.cc:240] Negotiation successful
0929 19:01:15.361378 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6236,"thread_start_us":489,"threads_started":1}
I20260929 19:01:15.361567 7254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:15.342082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34210 (local address 127.0.98.60:39419)
0929 19:01:15.342230 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:15.342231 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:15.342237 (+ 6us) server_negotiation.cc:407] Connection header received
0929 19:01:15.342279 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:15.342279 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:15.342286 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:15.342330 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:01:15.342658 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:15.343102 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:15.344041 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:15.344213 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:15.344257 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:15.345905 (+ 1648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:01:15.345906 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:01:15.345906 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:01:15.345913 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:01:15.359831 (+ 13918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:15.360262 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:15.360262 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:15.360263 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:15.360323 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:15.360554 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:15.360555 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:15.360555 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:15.360683 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:01:15.361245 (+ 562us) server_negotiation.cc:1035] Waiting for connection context
0929 19:01:15.361347 (+ 102us) server_negotiation.cc:299] Negotiation successful
0929 19:01:15.361442 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":88,"threads_started":1}
I20260929 19:01:15.362215 7253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:15.333299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42823 (local address 127.0.0.1:44980)
0929 19:01:15.333983 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:15.333992 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 19:01:15.334075 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:01:15.334284 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:01:15.334284 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:01:15.334453 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:01:15.334509 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:15.334510 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:15.335167 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:15.335167 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:15.335728 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:15.335730 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:15.336206 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:15.336206 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:15.336253 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:15.357395 (+ 21142us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:01:15.357400 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:01:15.358355 (+ 955us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:01:15.360385 (+ 2030us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:01:15.360385 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:01:15.360386 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:01:15.360592 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:01:15.360809 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:01:15.360810 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:01:15.360810 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:01:15.360892 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:01:15.361889 (+ 997us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:01:15.361889 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:01:15.361928 (+ 39us) client_negotiation.cc:769] Sending connection context
0929 19:01:15.361993 (+ 65us) client_negotiation.cc:240] Negotiation successful
0929 19:01:15.362104 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":669,"thread_start_us":70,"threads_started":1}
I20260929 19:01:15.362226 7158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:15.333357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44980 (local address 127.0.98.61:42823)
0929 19:01:15.333402 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:15.333404 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:15.334052 (+ 648us) server_negotiation.cc:407] Connection header received
0929 19:01:15.334139 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:15.334140 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:15.334149 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:15.334212 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:01:15.334592 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:15.335099 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:15.335810 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:15.336104 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:15.336139 (+ 35us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:15.358476 (+ 22337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:01:15.358478 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:01:15.358478 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:01:15.358486 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:01:15.360302 (+ 1816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:15.360668 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:15.360669 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:15.360669 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:15.360718 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:15.360964 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:15.360965 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:15.360965 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:15.361100 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:01:15.361143 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0929 19:01:15.362038 (+ 895us) server_negotiation.cc:299] Negotiation successful
0929 19:01:15.362138 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260929 19:01:15.362232 6509 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.60:39419: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "0908a5f3d25543efb7c1f184cd5178f4"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.60"
port: 39419
}
attrs {
promote: true
}
}
dest_uuid: "ea566150fd024ed9b9485efbb8ee914c"
cas_config_opid_index: 5
I20260929 19:01:15.362493 6509 consensus_queue.cc:237] T 00000000000000000000000000000000 P ea566150fd024ed9b9485efbb8ee914c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ea566150fd024ed9b9485efbb8ee914c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42245 } } peers { permanent_uuid: "f5eac47223df4bf997b0f24a92898e8f" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 42823 } attrs { promote: true } } peers { permanent_uuid: "0908a5f3d25543efb7c1f184cd5178f4" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39419 } attrs { promote: true } }
I20260929 19:01:15.362941 7069 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ea566150fd024ed9b9485efbb8ee914c [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 0908a5f3d25543efb7c1f184cd5178f4 (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ea566150fd024ed9b9485efbb8ee914c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42245 } } peers { permanent_uuid: "f5eac47223df4bf997b0f24a92898e8f" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 42823 } attrs { promote: true } } peers { permanent_uuid: "0908a5f3d25543efb7c1f184cd5178f4" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39419 } attrs { promote: true } } }
I20260929 19:01:15.363183 7203 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0908a5f3d25543efb7c1f184cd5178f4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:01:15.363690 7069 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.60:39419
I20260929 19:01:15.363773 7069 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea566150fd024ed9b9485efbb8ee914c [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "ea566150fd024ed9b9485efbb8ee914c" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ea566150fd024ed9b9485efbb8ee914c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42245 } } peers { permanent_uuid: "f5eac47223df4bf997b0f24a92898e8f" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 42823 } attrs { promote: true } } peers { permanent_uuid: "0908a5f3d25543efb7c1f184cd5178f4" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39419 } attrs { promote: true } } }
I20260929 19:01:15.363845 7069 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea566150fd024ed9b9485efbb8ee914c [sys.catalog]: This master's current role is: LEADER
I20260929 19:01:15.363929 7161 master.cc:537] Master@127.0.98.60:39419 shutting down...
I20260929 19:01:15.364059 7203 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0908a5f3d25543efb7c1f184cd5178f4 [term 1 FOLLOWER]: Refusing update from remote peer ea566150fd024ed9b9485efbb8ee914c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260929 19:01:15.364511 7068 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ea566150fd024ed9b9485efbb8ee914c [LEADER]: Connected to new peer: Peer: permanent_uuid: "0908a5f3d25543efb7c1f184cd5178f4" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39419 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 19:01:15.365324 6494 consensus_peers.cc:487] T 00000000000000000000000000000000 P ea566150fd024ed9b9485efbb8ee914c -> Peer 0908a5f3d25543efb7c1f184cd5178f4 (127.0.98.60:39419): Couldn't send request to peer 0908a5f3d25543efb7c1f184cd5178f4. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260929 19:01:15.367674 7161 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 0908a5f3d25543efb7c1f184cd5178f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 19:01:15.367700 7067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:15.362904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42823 (local address 127.0.0.1:44992)
0929 19:01:15.363051 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:15.363061 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 19:01:15.363101 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:01:15.364282 (+ 1181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:01:15.364283 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:01:15.364285 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:01:15.364343 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:15.364344 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:15.365319 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:15.365320 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:15.366522 (+ 1202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:15.366525 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:15.367407 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:15.367408 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:15.367484 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:15.367526 (+ 42us) client_negotiation.cc:769] Sending connection context
0929 19:01:15.367602 (+ 76us) client_negotiation.cc:240] Negotiation successful
0929 19:01:15.367636 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131}
I20260929 19:01:15.367949 7158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:15.363917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44992 (local address 127.0.98.61:42823)
0929 19:01:15.364106 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:15.364108 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:15.364116 (+ 8us) server_negotiation.cc:407] Connection header received
0929 19:01:15.364136 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:15.364137 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:15.364159 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:15.364222 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:01:15.364451 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:15.365223 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:15.366617 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:15.367225 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:15.367290 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:15.367374 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0929 19:01:15.367843 (+ 469us) server_negotiation.cc:299] Negotiation successful
0929 19:01:15.367878 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170}
I20260929 19:01:15.367959 7161 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 0908a5f3d25543efb7c1f184cd5178f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 19:01:15.368021 7161 tablet_replica.cc:330] T 00000000000000000000000000000000 P 0908a5f3d25543efb7c1f184cd5178f4: stopping tablet replica
I20260929 19:01:15.368273 7117 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f5eac47223df4bf997b0f24a92898e8f [term 1 LEARNER]: Refusing update from remote peer ea566150fd024ed9b9485efbb8ee914c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260929 19:01:15.368438 7068 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ea566150fd024ed9b9485efbb8ee914c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5eac47223df4bf997b0f24a92898e8f" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 42823 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260929 19:01:15.368981 7117 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f5eac47223df4bf997b0f24a92898e8f [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 0908a5f3d25543efb7c1f184cd5178f4 (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ea566150fd024ed9b9485efbb8ee914c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42245 } } peers { permanent_uuid: "f5eac47223df4bf997b0f24a92898e8f" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 42823 } attrs { promote: true } } peers { permanent_uuid: "0908a5f3d25543efb7c1f184cd5178f4" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39419 } attrs { promote: true } } }
I20260929 19:01:15.369999 7146 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5eac47223df4bf997b0f24a92898e8f [sys.catalog]: SysCatalogTable state changed. Reason: New leader ea566150fd024ed9b9485efbb8ee914c. Latest consensus state: current_term: 1 leader_uuid: "ea566150fd024ed9b9485efbb8ee914c" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ea566150fd024ed9b9485efbb8ee914c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42245 } } peers { permanent_uuid: "f5eac47223df4bf997b0f24a92898e8f" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 42823 } attrs { promote: true } } peers { permanent_uuid: "0908a5f3d25543efb7c1f184cd5178f4" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39419 } attrs { promote: true } } }
I20260929 19:01:15.370071 7146 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5eac47223df4bf997b0f24a92898e8f [sys.catalog]: This master's current role is: LEARNER
I20260929 19:01:15.370102 7146 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5eac47223df4bf997b0f24a92898e8f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ea566150fd024ed9b9485efbb8ee914c" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ea566150fd024ed9b9485efbb8ee914c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42245 } } peers { permanent_uuid: "f5eac47223df4bf997b0f24a92898e8f" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 42823 } attrs { promote: true } } peers { permanent_uuid: "0908a5f3d25543efb7c1f184cd5178f4" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39419 } attrs { promote: true } } }
I20260929 19:01:15.370147 7146 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5eac47223df4bf997b0f24a92898e8f [sys.catalog]: This master's current role is: LEARNER
I20260929 19:01:15.370376 7069 raft_consensus.cc:1050] T 00000000000000000000000000000000 P ea566150fd024ed9b9485efbb8ee914c: attempting to promote NON_VOTER f5eac47223df4bf997b0f24a92898e8f to VOTER
I20260929 19:01:15.370579 7069 consensus_queue.cc:237] T 00000000000000000000000000000000 P ea566150fd024ed9b9485efbb8ee914c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ea566150fd024ed9b9485efbb8ee914c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42245 } } peers { permanent_uuid: "f5eac47223df4bf997b0f24a92898e8f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42823 } attrs { promote: false } } peers { permanent_uuid: "0908a5f3d25543efb7c1f184cd5178f4" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39419 } attrs { promote: true } }
I20260929 19:01:15.371021 7117 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f5eac47223df4bf997b0f24a92898e8f [term 1 LEARNER]: Refusing update from remote peer ea566150fd024ed9b9485efbb8ee914c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20260929 19:01:15.371209 6494 consensus_peers.cc:487] T 00000000000000000000000000000000 P ea566150fd024ed9b9485efbb8ee914c -> Peer 0908a5f3d25543efb7c1f184cd5178f4 (127.0.98.60:39419): Couldn't send request to peer 0908a5f3d25543efb7c1f184cd5178f4. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260929 19:01:15.371275 7257 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ea566150fd024ed9b9485efbb8ee914c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5eac47223df4bf997b0f24a92898e8f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42823 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260929 19:01:15.371891 7068 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ea566150fd024ed9b9485efbb8ee914c [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, f5eac47223df4bf997b0f24a92898e8f (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ea566150fd024ed9b9485efbb8ee914c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42245 } } peers { permanent_uuid: "f5eac47223df4bf997b0f24a92898e8f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42823 } attrs { promote: false } } peers { permanent_uuid: "0908a5f3d25543efb7c1f184cd5178f4" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39419 } attrs { promote: true } } }
I20260929 19:01:15.372635 7068 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea566150fd024ed9b9485efbb8ee914c [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "ea566150fd024ed9b9485efbb8ee914c" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ea566150fd024ed9b9485efbb8ee914c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42245 } } peers { permanent_uuid: "f5eac47223df4bf997b0f24a92898e8f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42823 } attrs { promote: false } } peers { permanent_uuid: "0908a5f3d25543efb7c1f184cd5178f4" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39419 } attrs { promote: true } } }
I20260929 19:01:15.372646 7257 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea566150fd024ed9b9485efbb8ee914c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ea566150fd024ed9b9485efbb8ee914c" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ea566150fd024ed9b9485efbb8ee914c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42245 } } peers { permanent_uuid: "f5eac47223df4bf997b0f24a92898e8f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42823 } attrs { promote: false } } peers { permanent_uuid: "0908a5f3d25543efb7c1f184cd5178f4" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39419 } attrs { promote: true } } }
I20260929 19:01:15.372757 7257 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea566150fd024ed9b9485efbb8ee914c [sys.catalog]: This master's current role is: LEADER
I20260929 19:01:15.372844 7117 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f5eac47223df4bf997b0f24a92898e8f [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, f5eac47223df4bf997b0f24a92898e8f (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ea566150fd024ed9b9485efbb8ee914c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42245 } } peers { permanent_uuid: "f5eac47223df4bf997b0f24a92898e8f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42823 } attrs { promote: false } } peers { permanent_uuid: "0908a5f3d25543efb7c1f184cd5178f4" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39419 } attrs { promote: true } } }
I20260929 19:01:15.373703 7256 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5eac47223df4bf997b0f24a92898e8f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ea566150fd024ed9b9485efbb8ee914c" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ea566150fd024ed9b9485efbb8ee914c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42245 } } peers { permanent_uuid: "f5eac47223df4bf997b0f24a92898e8f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42823 } attrs { promote: false } } peers { permanent_uuid: "0908a5f3d25543efb7c1f184cd5178f4" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39419 } attrs { promote: true } } }
I20260929 19:01:15.372723 7068 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea566150fd024ed9b9485efbb8ee914c [sys.catalog]: This master's current role is: LEADER
I20260929 19:01:15.373867 7256 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5eac47223df4bf997b0f24a92898e8f [sys.catalog]: This master's current role is: FOLLOWER
I20260929 19:01:15.382532 7161 master.cc:552] Master@127.0.98.60:39419 shutdown complete.
I20260929 19:01:15.382589 7161 master_runner.cc:305] Clearing existing system tablet
I20260929 19:01:15.382710 7161 fs_manager.cc:359] Metadata directory not provided
I20260929 19:01:15.382767 7161 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:01:15.383338 7161 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.001s
I20260929 19:01:15.383831 7259 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:15.383984 7161 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:01:15.384049 7161 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:15.384075 7161 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/minicluster-data/master-2/wal
uuid: "0908a5f3d25543efb7c1f184cd5178f4"
format_stamp: "Formatted at 2026-09-29 19:01:15 on dist-test-slave-lqkv"
server_key: "a1b90761f6adc10e05f7d41cad3b3c4b"
server_key_iv: "ae7a2d872241e0bc1ac48334e26ec360"
server_key_version: "encryptionkey@0"
I20260929 19:01:15.384796 7161 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 0908a5f3d25543efb7c1f184cd5178f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 19:01:15.386078 7161 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 0908a5f3d25543efb7c1f184cd5178f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260929 19:01:15.386144 7161 log.cc:1196] T 00000000000000000000000000000000 P 0908a5f3d25543efb7c1f184cd5178f4: Deleting WAL directory at /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260929 19:01:15.386327 7161 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 0908a5f3d25543efb7c1f184cd5178f4: Deleting consensus metadata
I20260929 19:01:15.386443 7161 master_runner.cc:315] Copying system tablet from 127.0.98.62:42245
I20260929 19:01:15.386997 7161 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 0908a5f3d25543efb7c1f184cd5178f4: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:42245
I20260929 19:01:15.392939 6550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:15.387297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54620 (local address 127.0.98.62:42245)
0929 19:01:15.387341 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:15.387343 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:15.387457 (+ 114us) server_negotiation.cc:407] Connection header received
0929 19:01:15.387528 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:15.387529 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:15.387538 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:15.387594 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:01:15.387897 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:15.388381 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:15.389462 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:15.389652 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:15.389731 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:15.390605 (+ 874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:01:15.390607 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:01:15.390607 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:01:15.390613 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:01:15.391861 (+ 1248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:15.392201 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:15.392202 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:15.392202 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:15.392254 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:15.392446 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:15.392447 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:15.392448 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:15.392554 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:01:15.392625 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 19:01:15.392777 (+ 152us) server_negotiation.cc:299] Negotiation successful
0929 19:01:15.392857 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260929 19:01:15.392948 7265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:15.387260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42245 (local address 127.0.0.1:54620)
0929 19:01:15.387382 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:15.387391 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 19:01:15.387470 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:01:15.387600 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:01:15.387601 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:01:15.387767 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:01:15.387821 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:15.387822 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:15.388459 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:15.388460 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:15.389369 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:15.389371 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:15.389741 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:15.389741 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:15.389810 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:15.390247 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:01:15.390251 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:01:15.390511 (+ 260us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:01:15.391965 (+ 1454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:01:15.391966 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:01:15.391966 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:01:15.392130 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:01:15.392324 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:01:15.392324 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:01:15.392325 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:01:15.392378 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:01:15.392617 (+ 239us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:01:15.392618 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:01:15.392659 (+ 41us) client_negotiation.cc:769] Sending connection context
0929 19:01:15.392748 (+ 89us) client_negotiation.cc:240] Negotiation successful
0929 19:01:15.392859 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":68,"threads_started":1}
I20260929 19:01:15.393314 6530 tablet_copy_service.cc:140] P ea566150fd024ed9b9485efbb8ee914c: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 0908a5f3d25543efb7c1f184cd5178f4 ({username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:54620)
I20260929 19:01:15.393360 6530 tablet_copy_service.cc:161] P ea566150fd024ed9b9485efbb8ee914c: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 0908a5f3d25543efb7c1f184cd5178f4 at {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:54620: session id = 0908a5f3d25543efb7c1f184cd5178f4-00000000000000000000000000000000
I20260929 19:01:15.393759 6530 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P ea566150fd024ed9b9485efbb8ee914c: Tablet Copy: opened 0 blocks and 1 log segments
I20260929 19:01:15.394212 7161 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260929 19:01:15.395295 7161 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 0908a5f3d25543efb7c1f184cd5178f4: tablet copy: Starting download of 0 data blocks...
I20260929 19:01:15.395428 7161 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 0908a5f3d25543efb7c1f184cd5178f4: tablet copy: Starting download of 1 WAL segments...
I20260929 19:01:15.396103 7161 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 0908a5f3d25543efb7c1f184cd5178f4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260929 19:01:15.396961 6530 tablet_copy_service.cc:342] P ea566150fd024ed9b9485efbb8ee914c: Request end of tablet copy session 0908a5f3d25543efb7c1f184cd5178f4-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:54620
I20260929 19:01:15.397014 6530 tablet_copy_service.cc:433] P ea566150fd024ed9b9485efbb8ee914c: ending tablet copy session 0908a5f3d25543efb7c1f184cd5178f4-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 0908a5f3d25543efb7c1f184cd5178f4
I20260929 19:01:15.399650 7161 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:01:15.400753 7267 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:01:15.400810 7269 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:01:15.400969 7266 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:01:15.401196 7161 server_base.cc:1034] running on GCE node
I20260929 19:01:15.401286 7161 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:01:15.401418 7161 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:01:15.402570 7161 hybrid_clock.cc:648] HybridClock initialized: now 1790708475402551 us; error 28 us; skew 500 ppm
Sep 29 19:01:15 dist-test-slave-lqkv krb5kdc[6465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708475, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:01:15.403775 7161 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260929 19:01:15.404083 7161 webserver.cc:460] Webserver started at http://127.0.98.60:38769/ using document root <none> and password file <none>
I20260929 19:01:15.404213 7161 fs_manager.cc:359] Metadata directory not provided
I20260929 19:01:15.404248 7161 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:01:15.404922 7161 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:15.405460 7276 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:15.405606 7161 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:15.405652 7161 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/minicluster-data/master-2/wal
uuid: "0908a5f3d25543efb7c1f184cd5178f4"
format_stamp: "Formatted at 2026-09-29 19:01:15 on dist-test-slave-lqkv"
server_key: "a1b90761f6adc10e05f7d41cad3b3c4b"
server_key_iv: "ae7a2d872241e0bc1ac48334e26ec360"
server_key_version: "encryptionkey@0"
I20260929 19:01:15.405700 7161 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:01:15.418512 7161 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:01:15.419937 7161 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:01:15.423033 7161 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:39419
I20260929 19:01:15.423210 7328 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:39419 every 8 connection(s)
I20260929 19:01:15.423384 7161 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/minicluster-data/master-2/data/info.pb
I20260929 19:01:15.424068 7329 sys_catalog.cc:263] Verifying existing consensus state
I20260929 19:01:15.424470 7329 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0908a5f3d25543efb7c1f184cd5178f4: Bootstrap starting.
I20260929 19:01:15.426813 7329 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0908a5f3d25543efb7c1f184cd5178f4: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 19:01:15.427043 7329 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0908a5f3d25543efb7c1f184cd5178f4: Bootstrap complete.
I20260929 19:01:15.427237 7329 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0908a5f3d25543efb7c1f184cd5178f4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ea566150fd024ed9b9485efbb8ee914c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42245 } } peers { permanent_uuid: "f5eac47223df4bf997b0f24a92898e8f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42823 } attrs { promote: false } } peers { permanent_uuid: "0908a5f3d25543efb7c1f184cd5178f4" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39419 } attrs { promote: true } }
I20260929 19:01:15.427304 7329 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0908a5f3d25543efb7c1f184cd5178f4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0908a5f3d25543efb7c1f184cd5178f4, State: Initialized, Role: LEARNER
I20260929 19:01:15.427330 7329 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0908a5f3d25543efb7c1f184cd5178f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ea566150fd024ed9b9485efbb8ee914c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42245 } } peers { permanent_uuid: "f5eac47223df4bf997b0f24a92898e8f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42823 } attrs { promote: false } } peers { permanent_uuid: "0908a5f3d25543efb7c1f184cd5178f4" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39419 } attrs { promote: true } }
I20260929 19:01:15.427507 7329 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0908a5f3d25543efb7c1f184cd5178f4 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 19:01:15.427495 7332 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0908a5f3d25543efb7c1f184cd5178f4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ea566150fd024ed9b9485efbb8ee914c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42245 } } peers { permanent_uuid: "f5eac47223df4bf997b0f24a92898e8f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42823 } attrs { promote: false } } peers { permanent_uuid: "0908a5f3d25543efb7c1f184cd5178f4" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39419 } attrs { promote: true } } }
I20260929 19:01:15.427595 7332 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0908a5f3d25543efb7c1f184cd5178f4 [sys.catalog]: This master's current role is: LEARNER
I20260929 19:01:15.428957 7343 catalog_manager.cc:1259] Loaded cluster ID: 273b8d19129f4d25a0e7c884d5c7d851
I20260929 19:01:15.429003 7343 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0908a5f3d25543efb7c1f184cd5178f4: loading cluster ID for follower catalog manager: success
I20260929 19:01:15.430236 7343 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 0908a5f3d25543efb7c1f184cd5178f4: acquiring CA information for follower catalog manager: success
I20260929 19:01:15.430500 7343 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 0908a5f3d25543efb7c1f184cd5178f4: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 19:01:15.477864 7344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:15.471979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34230 (local address 127.0.98.60:39419)
0929 19:01:15.472111 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:15.472112 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:15.472121 (+ 9us) server_negotiation.cc:407] Connection header received
0929 19:01:15.472145 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:15.472145 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:15.472154 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:15.472213 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:01:15.472791 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:15.473519 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:15.474185 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:15.474365 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:15.474417 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:15.475492 (+ 1075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:01:15.475493 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:01:15.475494 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:01:15.475499 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:01:15.476852 (+ 1353us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:15.477219 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:15.477219 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:15.477220 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:15.477269 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:15.477488 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:15.477489 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:15.477489 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:15.477593 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:01:15.477667 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 19:01:15.477746 (+ 79us) server_negotiation.cc:299] Negotiation successful
0929 19:01:15.477799 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96,"thread_start_us":75,"threads_started":1}
I20260929 19:01:15.478379 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUbnLHE/build/release/bin/kudu with pid 6572
I20260929 19:01:15.482893 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUbnLHE/build/release/bin/kudu
/tmp/dist-test-taskUbnLHE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:46675
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:42245
--webserver_port=33741
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:42245,127.0.98.61:42823
--tserver_master_addrs=127.0.98.62:42245,127.0.98.61:42823,127.0.98.60:39419 with env {KRB5CCNAME=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:01:15.561532 7345 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:01:15.561718 7345 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:01:15.561740 7345 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:01:15.562831 7345 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:01:15.562871 7345 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:01:15.562923 7345 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 19:01:15.564352 7345 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:46675
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=33741
--tserver_master_addrs=127.0.98.62:42245,127.0.98.61:42823,127.0.98.60:39419
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7345
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 18:09:13 UTC on ccycloud.quasar-hpkalf.root.comops.site
build id 4680913
I20260929 19:01:15.564579 7345 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:01:15.564797 7345 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:01:15.569480 7351 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:01:15.569475 7353 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:01:15.569491 7350 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:01:15.569849 7345 server_base.cc:1034] running on GCE node
I20260929 19:01:15.570053 7345 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:01:15.570281 7345 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:01:15.571462 7345 hybrid_clock.cc:648] HybridClock initialized: now 1790708475571451 us; error 25 us; skew 500 ppm
Sep 29 19:01:15 dist-test-slave-lqkv krb5kdc[6465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708475, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:01:15.573979 7345 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260929 19:01:15.574404 7345 webserver.cc:460] Webserver started at http://127.0.98.1:33741/ using document root <none> and password file <none>
I20260929 19:01:15.574630 7345 fs_manager.cc:359] Metadata directory not provided
I20260929 19:01:15.574716 7345 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:01:15.575999 7345 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260929 19:01:15.576606 7360 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:15.576785 7345 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 19:01:15.576897 7345 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/minicluster-data/ts-0/wal
uuid: "137b431b75d748d2bbc3786ad9df1faa"
format_stamp: "Formatted at 2026-09-29 19:01:14 on dist-test-slave-lqkv"
server_key: "a73faf3300b87fcc63f35f825fa3e33b"
server_key_iv: "a5eab158b4fa48e75fc1cc5637052877"
server_key_version: "encryptionkey@0"
I20260929 19:01:15.577263 7345 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:01:15.591562 7345 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:01:15.593281 7345 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:01:15.593422 7345 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:01:15.593616 7345 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:01:15.593880 7345 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:01:15.593931 7345 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:15.593969 7345 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:01:15.593997 7345 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:15.599478 7345 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46675
I20260929 19:01:15.599522 7473 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46675 every 8 connection(s)
I20260929 19:01:15.599813 7345 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/minicluster-data/ts-0/data/info.pb
Sep 29 19:01:15 dist-test-slave-lqkv krb5kdc[6465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708475, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 29 19:01:15 dist-test-slave-lqkv krb5kdc[6465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708475, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260929 19:01:15.607155 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUbnLHE/build/release/bin/kudu as pid 7345
I20260929 19:01:15.607321 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUbnLHE/build/release/bin/kudu with pid 6706
Sep 29 19:01:15 dist-test-slave-lqkv krb5kdc[6465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708475, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260929 19:01:15.610831 7158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:15.601058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46337 (local address 127.0.98.61:42823)
0929 19:01:15.601100 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:15.601102 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:15.601180 (+ 78us) server_negotiation.cc:407] Connection header received
0929 19:01:15.601463 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:15.601464 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:15.601477 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:15.601509 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:01:15.602419 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:15.603141 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:15.604215 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:15.604504 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:15.604564 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:15.607046 (+ 2482us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:01:15.607048 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:01:15.607048 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:01:15.607056 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:01:15.609251 (+ 2195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:15.609722 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:15.609723 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:15.609723 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:15.609778 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:15.610106 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:15.610107 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:15.610107 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:15.610254 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:01:15.610298 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0929 19:01:15.610660 (+ 362us) server_negotiation.cc:299] Negotiation successful
0929 19:01:15.610730 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260929 19:01:15.611037 6550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:15.600916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48559 (local address 127.0.98.62:42245)
0929 19:01:15.600956 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:15.600958 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:15.600980 (+ 22us) server_negotiation.cc:407] Connection header received
0929 19:01:15.601449 (+ 469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:15.601450 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:15.601462 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:15.601519 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:01:15.602404 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:15.602878 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:15.603809 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:15.603991 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:15.604072 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:15.606531 (+ 2459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:01:15.606533 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:01:15.606533 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:01:15.606540 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:01:15.609689 (+ 3149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:15.610220 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:15.610221 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:15.610222 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:15.610276 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:15.610453 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:15.610453 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:15.610454 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:15.610580 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:01:15.610618 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0929 19:01:15.610818 (+ 200us) server_negotiation.cc:299] Negotiation successful
0929 19:01:15.610898 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260929 19:01:15.611378 7480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:15.600911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42823 (local address 127.0.98.1:46337)
0929 19:01:15.601113 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:15.601139 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0929 19:01:15.601360 (+ 221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:01:15.601597 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:01:15.601598 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:01:15.602128 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:01:15.602315 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:15.602321 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:15.603236 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:15.603237 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:15.604137 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:15.604139 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:15.604581 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:15.604582 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:15.604674 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:15.605086 (+ 412us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:01:15.605091 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:01:15.606928 (+ 1837us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:01:15.609337 (+ 2409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:01:15.609343 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:01:15.609345 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:01:15.609632 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:01:15.609853 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:01:15.609854 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:01:15.609854 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:01:15.610023 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:01:15.610347 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:01:15.610353 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:01:15.610500 (+ 147us) client_negotiation.cc:769] Sending connection context
0929 19:01:15.610955 (+ 455us) client_negotiation.cc:240] Negotiation successful
0929 19:01:15.611126 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"mutex_wait_us":31,"thread_start_us":55,"threads_started":1}
I20260929 19:01:15.611922 7479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:15.600739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42245 (local address 127.0.98.1:48559)
0929 19:01:15.600866 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:15.600877 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 19:01:15.601360 (+ 483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:01:15.601854 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:01:15.601855 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:01:15.602111 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:01:15.602314 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:15.602318 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:15.602976 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:15.602977 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:15.603721 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:15.603723 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:15.604078 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:15.604079 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:15.604295 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:15.604856 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:01:15.604864 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:01:15.606423 (+ 1559us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:01:15.609838 (+ 3415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:01:15.609839 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:01:15.609840 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:01:15.610139 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:01:15.610340 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:01:15.610340 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:01:15.610341 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:01:15.610394 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:01:15.610671 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:01:15.610672 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:01:15.610713 (+ 41us) client_negotiation.cc:769] Sending connection context
0929 19:01:15.611586 (+ 873us) client_negotiation.cc:240] Negotiation successful
0929 19:01:15.611722 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"mutex_wait_us":15,"thread_start_us":87,"threads_started":1}
I20260929 19:01:15.613018 7344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:15.600365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42285 (local address 127.0.98.60:39419)
0929 19:01:15.600406 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:15.600408 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:15.600732 (+ 324us) server_negotiation.cc:407] Connection header received
0929 19:01:15.601476 (+ 744us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:15.601477 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:15.601492 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:15.601522 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:01:15.602425 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:15.603144 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:15.604607 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:15.604940 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:15.605006 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:15.610162 (+ 5156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:01:15.610163 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:01:15.610164 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:01:15.610171 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:01:15.611783 (+ 1612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:15.612229 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:15.612231 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:15.612231 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:15.612292 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:15.612483 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:15.612485 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:15.612485 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:15.612600 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:01:15.612669 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0929 19:01:15.612850 (+ 181us) server_negotiation.cc:299] Negotiation successful
0929 19:01:15.612925 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260929 19:01:15.613026 7477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:15.600373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39419 (local address 127.0.98.1:42285)
0929 19:01:15.600612 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:15.600658 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 19:01:15.601351 (+ 693us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:01:15.601589 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:01:15.601594 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:01:15.602118 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:01:15.602315 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:15.602321 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:15.603236 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:15.603237 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:15.604136 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:15.604139 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:15.605057 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:15.605058 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:15.605115 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:15.605518 (+ 403us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:01:15.605523 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:01:15.610077 (+ 4554us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:01:15.611868 (+ 1791us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:01:15.611869 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:01:15.611869 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:01:15.612150 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:01:15.612361 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:01:15.612362 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:01:15.612362 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:01:15.612411 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:01:15.612685 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:01:15.612686 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:01:15.612724 (+ 38us) client_negotiation.cc:769] Sending connection context
0929 19:01:15.612805 (+ 81us) client_negotiation.cc:240] Negotiation successful
0929 19:01:15.612916 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"mutex_wait_us":60,"thread_start_us":172,"threads_started":1}
I20260929 19:01:15.613158 7476 heartbeater.cc:347] Connected to a master server at 127.0.98.61:42823
I20260929 19:01:15.613247 7476 heartbeater.cc:464] Registering TS with master...
I20260929 19:01:15.613284 7475 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42245
I20260929 19:01:15.613322 7475 heartbeater.cc:464] Registering TS with master...
I20260929 19:01:15.613415 7476 heartbeater.cc:511] Master 127.0.98.61:42823 requested a full tablet report, sending...
I20260929 19:01:15.613415 7475 heartbeater.cc:511] Master 127.0.98.62:42245 requested a full tablet report, sending...
I20260929 19:01:15.613726 7474 heartbeater.cc:347] Connected to a master server at 127.0.98.60:39419
I20260929 19:01:15.613780 7474 heartbeater.cc:464] Registering TS with master...
I20260929 19:01:15.613835 7474 heartbeater.cc:511] Master 127.0.98.60:39419 requested a full tablet report, sending...
I20260929 19:01:15.613868 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUbnLHE/build/release/bin/kudu
/tmp/dist-test-taskUbnLHE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:35351
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:42245
--webserver_port=46503
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:42245,127.0.98.61:42823
--tserver_master_addrs=127.0.98.62:42245,127.0.98.61:42823,127.0.98.60:39419 with env {KRB5CCNAME=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 19:01:15.613909 7107 ts_manager.cc:194] Registered new tserver with Master: 137b431b75d748d2bbc3786ad9df1faa (127.0.98.1:46675)
I20260929 19:01:15.613969 6509 ts_manager.cc:194] Re-registered known tserver with Master: 137b431b75d748d2bbc3786ad9df1faa (127.0.98.1:46675)
I20260929 19:01:15.614337 6509 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:48559
I20260929 19:01:15.614737 7293 ts_manager.cc:194] Registered new tserver with Master: 137b431b75d748d2bbc3786ad9df1faa (127.0.98.1:46675)
W20260929 19:01:15.694664 7481 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:01:15.694840 7481 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:01:15.694861 7481 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:01:15.695897 7481 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:01:15.695928 7481 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:01:15.695953 7481 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 19:01:15.697335 7481 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:35351
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=46503
--tserver_master_addrs=127.0.98.62:42245,127.0.98.61:42823,127.0.98.60:39419
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7481
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 18:09:13 UTC on ccycloud.quasar-hpkalf.root.comops.site
build id 4680913
I20260929 19:01:15.697499 7481 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:01:15.697655 7481 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:01:15.702291 7487 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:01:15.702301 7486 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:01:15.702301 7489 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:01:15.702543 7481 server_base.cc:1034] running on GCE node
I20260929 19:01:15.702746 7481 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:01:15.702920 7481 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:01:15.704047 7481 hybrid_clock.cc:648] HybridClock initialized: now 1790708475704037 us; error 27 us; skew 500 ppm
Sep 29 19:01:15 dist-test-slave-lqkv krb5kdc[6465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708475, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:01:15.706737 7481 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260929 19:01:15.707211 7481 webserver.cc:460] Webserver started at http://127.0.98.2:46503/ using document root <none> and password file <none>
I20260929 19:01:15.707445 7481 fs_manager.cc:359] Metadata directory not provided
I20260929 19:01:15.707525 7481 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:01:15.708777 7481 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 19:01:15.709338 7496 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:15.709512 7481 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:15.709604 7481 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/minicluster-data/ts-1/wal
uuid: "84528791e3a442dda71a633e7a3b6140"
format_stamp: "Formatted at 2026-09-29 19:01:14 on dist-test-slave-lqkv"
server_key: "420b359ac04448c53dcbe214dcb7df5c"
server_key_iv: "b124fdffbcfb3b4ed59faa37ab6df22a"
server_key_version: "encryptionkey@0"
I20260929 19:01:15.709904 7481 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:01:15.717955 7481 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:01:15.719619 7481 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:01:15.719774 7481 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:01:15.719971 7481 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:01:15.720252 7481 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:01:15.720305 7481 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:15.720345 7481 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:01:15.720372 7481 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:15.726369 7481 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35351
I20260929 19:01:15.726414 7609 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35351 every 8 connection(s)
I20260929 19:01:15.726719 7481 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/minicluster-data/ts-1/data/info.pb
I20260929 19:01:15.728211 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUbnLHE/build/release/bin/kudu as pid 7481
I20260929 19:01:15.728370 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUbnLHE/build/release/bin/kudu with pid 6840
I20260929 19:01:15.734376 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUbnLHE/build/release/bin/kudu
/tmp/dist-test-taskUbnLHE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:42589
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:42245
--webserver_port=33635
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:42245,127.0.98.61:42823
--tserver_master_addrs=127.0.98.62:42245,127.0.98.61:42823,127.0.98.60:39419 with env {KRB5CCNAME=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 19:01:15 dist-test-slave-lqkv krb5kdc[6465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708475, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Sep 29 19:01:15 dist-test-slave-lqkv krb5kdc[6465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708475, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 29 19:01:15 dist-test-slave-lqkv krb5kdc[6465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708475, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260929 19:01:15.739838 7344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:15.727477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57235 (local address 127.0.98.60:39419)
0929 19:01:15.727544 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:15.727547 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:15.728203 (+ 656us) server_negotiation.cc:407] Connection header received
0929 19:01:15.729963 (+ 1760us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:15.729964 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:15.729977 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:15.730009 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:01:15.730924 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:15.731415 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:15.732410 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:15.732630 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:15.732718 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:15.736003 (+ 3285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:01:15.736005 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:01:15.736005 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:01:15.736013 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:01:15.738156 (+ 2143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:15.738642 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:15.738643 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:15.738644 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:15.738731 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:15.739030 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:15.739031 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:15.739032 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:15.739169 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:01:15.739211 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0929 19:01:15.739606 (+ 395us) server_negotiation.cc:299] Negotiation successful
0929 19:01:15.739697 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260929 19:01:15.740177 7616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:15.727709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42245 (local address 127.0.98.2:50935)
0929 19:01:15.728063 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:15.728114 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 19:01:15.729858 (+ 1744us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:01:15.730172 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:01:15.730176 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:01:15.730652 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:01:15.730829 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:15.730835 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:15.731866 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:15.731867 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:15.732883 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:15.732885 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:15.733415 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:15.733416 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:15.733487 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:15.733906 (+ 419us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:01:15.733911 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:01:15.735819 (+ 1908us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:01:15.738173 (+ 2354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:01:15.738179 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:01:15.738181 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:01:15.738799 (+ 618us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:01:15.739012 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:01:15.739013 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:01:15.739013 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:01:15.739069 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:01:15.739645 (+ 576us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:01:15.739646 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:01:15.739686 (+ 40us) client_negotiation.cc:769] Sending connection context
0929 19:01:15.739773 (+ 87us) client_negotiation.cc:240] Negotiation successful
0929 19:01:15.739946 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":338,"mutex_wait_us":23,"thread_start_us":68,"threads_started":1}
I20260929 19:01:15.740180 7615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:15.727441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39419 (local address 127.0.98.2:57235)
0929 19:01:15.727932 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:15.728114 (+ 182us) client_negotiation.cc:174] Beginning negotiation
0929 19:01:15.729847 (+ 1733us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:01:15.730174 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:01:15.730181 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:01:15.730661 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:01:15.730829 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:15.730833 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:15.731542 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:15.731543 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:15.732320 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:15.732322 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:15.732727 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:15.732728 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:15.732930 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:15.733427 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:01:15.733436 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:01:15.735891 (+ 2455us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:01:15.738237 (+ 2346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:01:15.738238 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:01:15.738239 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:01:15.738528 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:01:15.738815 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:01:15.738815 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:01:15.738816 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:01:15.738947 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:01:15.739256 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:01:15.739262 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:01:15.739450 (+ 188us) client_negotiation.cc:769] Sending connection context
0929 19:01:15.739986 (+ 536us) client_negotiation.cc:240] Negotiation successful
0929 19:01:15.740064 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":372,"mutex_wait_us":1,"thread_start_us":76,"threads_started":1}
I20260929 19:01:15.740675 7610 heartbeater.cc:347] Connected to a master server at 127.0.98.60:39419
I20260929 19:01:15.740769 7610 heartbeater.cc:464] Registering TS with master...
I20260929 19:01:15.740952 7610 heartbeater.cc:511] Master 127.0.98.60:39419 requested a full tablet report, sending...
I20260929 19:01:15.741078 6550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:15.727805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50935 (local address 127.0.98.62:42245)
0929 19:01:15.727851 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:15.727852 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:15.728201 (+ 349us) server_negotiation.cc:407] Connection header received
0929 19:01:15.729967 (+ 1766us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:15.729969 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:15.729981 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:15.730010 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:01:15.730933 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:15.731771 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:15.732979 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:15.733295 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:15.733338 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:15.735906 (+ 2568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:01:15.735907 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:01:15.735908 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:01:15.735915 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:01:15.738084 (+ 2169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:15.738888 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:15.738889 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:15.738889 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:15.738942 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:15.739390 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:15.739392 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:15.739392 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:15.739549 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:01:15.739608 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0929 19:01:15.740910 (+ 1302us) server_negotiation.cc:299] Negotiation successful
0929 19:01:15.740993 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260929 19:01:15.741444 7293 ts_manager.cc:194] Registered new tserver with Master: 84528791e3a442dda71a633e7a3b6140 (127.0.98.2:35351)
I20260929 19:01:15.741659 7158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:15.727947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35555 (local address 127.0.98.61:42823)
0929 19:01:15.728014 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:15.728016 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:15.733332 (+ 5316us) server_negotiation.cc:407] Connection header received
0929 19:01:15.733452 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:15.733453 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:15.733465 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:15.733497 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:01:15.733952 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:15.734697 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:15.735717 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:15.736009 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:15.736051 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:15.738245 (+ 2194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:01:15.738246 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:01:15.738247 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:01:15.738254 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:01:15.740112 (+ 1858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:15.740559 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:15.740560 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:15.740561 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:15.740632 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:15.740893 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:15.740894 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:15.740894 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:15.741039 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:01:15.741085 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0929 19:01:15.741430 (+ 345us) server_negotiation.cc:299] Negotiation successful
0929 19:01:15.741516 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 19:01:15.741755 7611 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42245
I20260929 19:01:15.741812 7611 heartbeater.cc:464] Registering TS with master...
I20260929 19:01:15.741880 7611 heartbeater.cc:511] Master 127.0.98.62:42245 requested a full tablet report, sending...
I20260929 19:01:15.742168 6509 ts_manager.cc:194] Re-registered known tserver with Master: 84528791e3a442dda71a633e7a3b6140 (127.0.98.2:35351)
I20260929 19:01:15.742700 6509 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:50935
I20260929 19:01:15.743839 7614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:15.727870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42823 (local address 127.0.98.2:35555)
0929 19:01:15.728337 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:15.728347 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 19:01:15.733365 (+ 5018us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:01:15.733563 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:01:15.733563 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:01:15.733802 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:01:15.733860 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:15.733862 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:15.734799 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:15.734800 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:15.735618 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:15.735621 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:15.736130 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:15.736131 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:15.736201 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:15.736655 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:01:15.736661 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:01:15.738145 (+ 1484us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:01:15.740236 (+ 2091us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:01:15.740237 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:01:15.740238 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:01:15.740467 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:01:15.740713 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:01:15.740714 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:01:15.740714 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:01:15.740776 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:01:15.741138 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:01:15.741139 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:01:15.741185 (+ 46us) client_negotiation.cc:769] Sending connection context
0929 19:01:15.742052 (+ 867us) client_negotiation.cc:240] Negotiation successful
0929 19:01:15.743461 (+ 1409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":55,"threads_started":1}
I20260929 19:01:15.744212 7612 heartbeater.cc:347] Connected to a master server at 127.0.98.61:42823
I20260929 19:01:15.744271 7612 heartbeater.cc:464] Registering TS with master...
I20260929 19:01:15.744338 7612 heartbeater.cc:511] Master 127.0.98.61:42823 requested a full tablet report, sending...
I20260929 19:01:15.744585 7107 ts_manager.cc:194] Registered new tserver with Master: 84528791e3a442dda71a633e7a3b6140 (127.0.98.2:35351)
W20260929 19:01:15.816852 7617 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:01:15.817065 7617 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:01:15.817098 7617 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:01:15.818214 7617 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:01:15.818275 7617 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:01:15.818320 7617 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 19:01:15.819721 7617 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:42589
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=33635
--tserver_master_addrs=127.0.98.62:42245,127.0.98.61:42823,127.0.98.60:39419
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7617
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 18:09:13 UTC on ccycloud.quasar-hpkalf.root.comops.site
build id 4680913
I20260929 19:01:15.819965 7617 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:01:15.820170 7617 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:01:15.825071 7625 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:01:15.825098 7617 server_base.cc:1034] running on GCE node
W20260929 19:01:15.825071 7623 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:01:15.825062 7622 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:01:15.825519 7617 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:01:15.825800 7617 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:01:15.826957 7617 hybrid_clock.cc:648] HybridClock initialized: now 1790708475826950 us; error 34 us; skew 500 ppm
Sep 29 19:01:15 dist-test-slave-lqkv krb5kdc[6465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708475, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:01:15.829598 7617 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260929 19:01:15.829993 7617 webserver.cc:460] Webserver started at http://127.0.98.3:33635/ using document root <none> and password file <none>
I20260929 19:01:15.830209 7617 fs_manager.cc:359] Metadata directory not provided
I20260929 19:01:15.830282 7617 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:01:15.831459 7617 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260929 19:01:15.831974 7632 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:15.832141 7617 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 19:01:15.832235 7617 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/minicluster-data/ts-2/wal
uuid: "07b2d889f6074dcf998c9ef656c568ca"
format_stamp: "Formatted at 2026-09-29 19:01:14 on dist-test-slave-lqkv"
server_key: "7892eb957b9eb97562bf017f9e6b3843"
server_key_iv: "6eff6dbc37ea0201a8b50bdc01fcbfc6"
server_key_version: "encryptionkey@0"
I20260929 19:01:15.832549 7617 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:01:15.856379 7617 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:01:15.858724 7617 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:01:15.858877 7617 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:01:15.859079 7617 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:01:15.859351 7617 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:01:15.859400 7617 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:15.859437 7617 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:01:15.859463 7617 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:15.864749 7617 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42589
I20260929 19:01:15.864792 7745 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42589 every 8 connection(s)
I20260929 19:01:15.865175 7617 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/minicluster-data/ts-2/data/info.pb
I20260929 19:01:15.869656 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUbnLHE/build/release/bin/kudu as pid 7617
Sep 29 19:01:15 dist-test-slave-lqkv krb5kdc[6465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708475, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 29 19:01:15 dist-test-slave-lqkv krb5kdc[6465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708475, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Sep 29 19:01:15 dist-test-slave-lqkv krb5kdc[6465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708475, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260929 19:01:15.878000 6550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:15.866122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38881 (local address 127.0.98.62:42245)
0929 19:01:15.866164 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:15.866166 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:15.866219 (+ 53us) server_negotiation.cc:407] Connection header received
0929 19:01:15.866817 (+ 598us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:15.866818 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:15.866831 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:15.866889 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:01:15.867558 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:15.868383 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:15.869616 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:15.869941 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:15.869980 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:15.874149 (+ 4169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:01:15.874150 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:01:15.874150 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:01:15.874157 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:01:15.876276 (+ 2119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:15.876770 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:15.876771 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:15.876772 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:15.876866 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:15.877144 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:15.877145 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:15.877146 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:15.877286 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:01:15.877348 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0929 19:01:15.877818 (+ 470us) server_negotiation.cc:299] Negotiation successful
0929 19:01:15.877893 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260929 19:01:15.878415 7751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:15.866061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42245 (local address 127.0.98.3:38881)
0929 19:01:15.866158 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:15.866167 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 19:01:15.866714 (+ 547us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:01:15.866959 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:01:15.866960 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:01:15.867290 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:01:15.867465 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:15.867468 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:15.868649 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:15.868650 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:15.869511 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:15.869513 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:15.870052 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:15.870053 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:15.870118 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:15.870746 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:01:15.870755 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:01:15.874037 (+ 3282us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:01:15.876363 (+ 2326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:01:15.876369 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:01:15.876371 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:01:15.876685 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:01:15.876941 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:01:15.876942 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:01:15.876943 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:01:15.877068 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:01:15.877376 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:01:15.877382 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:01:15.877534 (+ 152us) client_negotiation.cc:769] Sending connection context
0929 19:01:15.877643 (+ 109us) client_negotiation.cc:240] Negotiation successful
0929 19:01:15.878173 (+ 530us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":79,"mutex_wait_us":5,"thread_start_us":58,"threads_started":1}
I20260929 19:01:15.879067 7747 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42245
I20260929 19:01:15.879153 7747 heartbeater.cc:464] Registering TS with master...
I20260929 19:01:15.879338 7747 heartbeater.cc:511] Master 127.0.98.62:42245 requested a full tablet report, sending...
I20260929 19:01:15.879375 7158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:15.865805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43587 (local address 127.0.98.61:42823)
0929 19:01:15.865858 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:15.865860 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:15.866061 (+ 201us) server_negotiation.cc:407] Connection header received
0929 19:01:15.866826 (+ 765us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:15.866828 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:15.866840 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:15.866873 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:01:15.867575 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:15.868269 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:15.869615 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:15.869897 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:15.869937 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:15.874330 (+ 4393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:01:15.874332 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:01:15.874332 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:01:15.874339 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:01:15.877084 (+ 2745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:15.878467 (+ 1383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:15.878468 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:15.878468 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:15.878521 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:15.878698 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:15.878698 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:15.878699 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:15.878820 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:01:15.878876 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0929 19:01:15.879204 (+ 328us) server_negotiation.cc:299] Negotiation successful
0929 19:01:15.879266 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260929 19:01:15.879684 6509 ts_manager.cc:194] Re-registered known tserver with Master: 07b2d889f6074dcf998c9ef656c568ca (127.0.98.3:42589)
I20260929 19:01:15.879743 7750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:15.865808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42823 (local address 127.0.98.3:43587)
0929 19:01:15.865946 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:15.865977 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0929 19:01:15.866714 (+ 737us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:01:15.866955 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:01:15.866959 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:01:15.867298 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:01:15.867465 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:15.867469 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:15.868377 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:15.868377 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:15.869514 (+ 1137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:15.869515 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:15.870074 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:15.870075 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:15.870126 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:15.870948 (+ 822us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:01:15.870953 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:01:15.874139 (+ 3186us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:01:15.877578 (+ 3439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:01:15.877579 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:01:15.877579 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:01:15.878182 (+ 603us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:01:15.878576 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:01:15.878576 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:01:15.878577 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:01:15.878631 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:01:15.878901 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:01:15.878902 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:01:15.878946 (+ 44us) client_negotiation.cc:769] Sending connection context
0929 19:01:15.879013 (+ 67us) client_negotiation.cc:240] Negotiation successful
0929 19:01:15.879096 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"mutex_wait_us":44,"thread_start_us":85,"threads_started":1}
I20260929 19:01:15.880159 6509 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:38881
I20260929 19:01:15.880198 7748 heartbeater.cc:347] Connected to a master server at 127.0.98.61:42823
I20260929 19:01:15.880237 7748 heartbeater.cc:464] Registering TS with master...
I20260929 19:01:15.880311 7748 heartbeater.cc:511] Master 127.0.98.61:42823 requested a full tablet report, sending...
I20260929 19:01:15.880698 7107 ts_manager.cc:194] Registered new tserver with Master: 07b2d889f6074dcf998c9ef656c568ca (127.0.98.3:42589)
I20260929 19:01:15.881405 7752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:15.866212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39419 (local address 127.0.98.3:53637)
0929 19:01:15.866356 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:15.866365 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 19:01:15.866714 (+ 349us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:01:15.867328 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:01:15.867329 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:01:15.867464 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:01:15.867517 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:15.867518 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:15.868463 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:15.868463 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:15.869346 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:15.869348 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:15.869953 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:15.869954 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:15.870082 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:15.870813 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:01:15.870818 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:01:15.873392 (+ 2574us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:01:15.879379 (+ 5987us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:01:15.879380 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:01:15.879380 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:01:15.879611 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:01:15.879870 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:01:15.879871 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:01:15.879871 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:01:15.879938 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:01:15.880910 (+ 972us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:01:15.880911 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:01:15.880951 (+ 40us) client_negotiation.cc:769] Sending connection context
0929 19:01:15.881023 (+ 72us) client_negotiation.cc:240] Negotiation successful
0929 19:01:15.881126 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"mutex_wait_us":39,"thread_start_us":72,"threads_started":1}
I20260929 19:01:15.881686 7344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:15.866267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53637 (local address 127.0.98.60:39419)
0929 19:01:15.866306 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:15.866308 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:15.866430 (+ 122us) server_negotiation.cc:407] Connection header received
0929 19:01:15.866831 (+ 401us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:15.866832 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:15.866844 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:15.866874 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:01:15.867612 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:15.868381 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:15.869512 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:15.869812 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:15.869882 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:15.873520 (+ 3638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:01:15.873522 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:01:15.873522 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:01:15.873528 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:01:15.879294 (+ 5766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:15.879737 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:15.879739 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:15.879739 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:15.879795 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:15.880384 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:15.880385 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:15.880385 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:15.880504 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:01:15.880570 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0929 19:01:15.881561 (+ 991us) server_negotiation.cc:299] Negotiation successful
0929 19:01:15.881630 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260929 19:01:15.881978 7746 heartbeater.cc:347] Connected to a master server at 127.0.98.60:39419
I20260929 19:01:15.882033 7746 heartbeater.cc:464] Registering TS with master...
I20260929 19:01:15.882102 7746 heartbeater.cc:511] Master 127.0.98.60:39419 requested a full tablet report, sending...
I20260929 19:01:15.882395 7293 ts_manager.cc:194] Registered new tserver with Master: 07b2d889f6074dcf998c9ef656c568ca (127.0.98.3:42589)
I20260929 19:01:15.883189 7757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:15.870754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54636 (local address 127.0.98.62:42245)
0929 19:01:15.870865 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:15.870867 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:15.871622 (+ 755us) server_negotiation.cc:407] Connection header received
0929 19:01:15.871723 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:15.871724 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:15.871737 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:15.871786 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:01:15.872668 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:15.873496 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:15.874688 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:15.874997 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:15.875033 (+ 36us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:15.876899 (+ 1866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:01:15.876900 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:01:15.876900 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:01:15.876907 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:01:15.881250 (+ 4343us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:15.882386 (+ 1136us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:15.882387 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:15.882388 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:15.882439 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:15.882635 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:15.882637 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:15.882637 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:15.882771 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:01:15.882888 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0929 19:01:15.882995 (+ 107us) server_negotiation.cc:299] Negotiation successful
0929 19:01:15.883072 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95,"thread_start_us":60,"threads_started":1}
I20260929 19:01:15.883740 7761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:15.871673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34242 (local address 127.0.98.60:39419)
0929 19:01:15.871779 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:15.871780 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:15.871987 (+ 207us) server_negotiation.cc:407] Connection header received
0929 19:01:15.872081 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:15.872082 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:15.872094 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:15.872134 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:01:15.872828 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:15.873554 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:15.874629 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:15.874928 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:15.874963 (+ 35us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:15.876899 (+ 1936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:01:15.876900 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:01:15.876900 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:01:15.876907 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:01:15.878706 (+ 1799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:15.882255 (+ 3549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:15.882256 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:15.882256 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:15.882314 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:15.882683 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:15.882684 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:15.882684 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:15.882794 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:01:15.882903 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 19:01:15.883541 (+ 638us) server_negotiation.cc:299] Negotiation successful
0929 19:01:15.883640 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86,"thread_start_us":59,"threads_started":1}
I20260929 19:01:15.885468 7759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:15.871371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45002 (local address 127.0.98.61:42823)
0929 19:01:15.871889 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:15.871891 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:15.871943 (+ 52us) server_negotiation.cc:407] Connection header received
0929 19:01:15.872050 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:15.872050 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:15.872061 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:15.872104 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:01:15.874770 (+ 2666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:15.879282 (+ 4512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:15.880265 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:15.880567 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:15.880602 (+ 35us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:15.882337 (+ 1735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:01:15.882338 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:01:15.882339 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:01:15.882345 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:01:15.884399 (+ 2054us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:15.884749 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:15.884750 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:15.884750 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:15.884823 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:15.885006 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:15.885007 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:15.885007 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:15.885110 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:01:15.885200 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0929 19:01:15.885306 (+ 106us) server_negotiation.cc:299] Negotiation successful
0929 19:01:15.885374 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":498,"thread_start_us":44,"threads_started":1}
I20260929 19:01:15.884678 6509 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54636:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 19:01:15.886402 6509 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 19:01:15.892508 7769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:15.889617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52252 (local address 127.0.98.3:42589)
0929 19:01:15.889760 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:15.889764 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:15.889902 (+ 138us) server_negotiation.cc:407] Connection header received
0929 19:01:15.889938 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:15.889941 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:15.889954 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:15.890030 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:01:15.890145 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:15.890897 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:15.891719 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:15.892156 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:15.892267 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:15.892344 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0929 19:01:15.892411 (+ 67us) server_negotiation.cc:299] Negotiation successful
0929 19:01:15.892440 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":85,"threads_started":1}
I20260929 19:01:15.892504 7768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:15.889523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42589 (local address 127.0.0.1:52252)
0929 19:01:15.889763 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:15.889782 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 19:01:15.889843 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:01:15.890027 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:01:15.890028 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:01:15.890029 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:01:15.890065 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:15.890066 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:15.890973 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:15.890974 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:15.891637 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:15.891639 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:15.892272 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:15.892273 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:15.892334 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:15.892340 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 19:01:15.892399 (+ 59us) client_negotiation.cc:240] Negotiation successful
0929 19:01:15.892439 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":44,"threads_started":1}
I20260929 19:01:15.893241 7680 tablet_service.cc:1467] Processing CreateTablet for tablet 0b2218d904bb45e98320b97c6b7d0a87 (DEFAULT_TABLE table=test-table [id=6995aade3cc540c280df3dcd1ee3609b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:01:15.893580 7680 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b2218d904bb45e98320b97c6b7d0a87. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:01:15.893779 7772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:15.889809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46675 (local address 127.0.0.1:33890)
0929 19:01:15.889907 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:15.889917 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 19:01:15.889966 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:01:15.890307 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:01:15.890308 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:01:15.890310 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:01:15.890343 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:15.890344 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:15.891558 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:15.891559 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:15.892677 (+ 1118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:15.892680 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:15.893525 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:15.893526 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:15.893582 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:15.893588 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 19:01:15.893681 (+ 93us) client_negotiation.cc:240] Negotiation successful
0929 19:01:15.893718 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":79,"thread_start_us":60,"threads_started":1}
I20260929 19:01:15.894063 7773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:15.890017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33890 (local address 127.0.98.1:46675)
0929 19:01:15.890154 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:15.890158 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:15.890166 (+ 8us) server_negotiation.cc:407] Connection header received
0929 19:01:15.890204 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:15.890206 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:15.890220 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:15.890298 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:01:15.890406 (+ 108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:15.891470 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:15.892763 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:15.893451 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:15.893526 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:15.893597 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 19:01:15.893934 (+ 337us) server_negotiation.cc:299] Negotiation successful
0929 19:01:15.893978 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":77,"threads_started":1}
I20260929 19:01:15.894098 7770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:15.889632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35351 (local address 127.0.0.1:49186)
0929 19:01:15.889811 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:15.889820 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 19:01:15.889860 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:01:15.890609 (+ 749us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:01:15.890610 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:01:15.890612 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:01:15.890648 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:15.890650 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:15.891820 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:15.891821 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:15.892909 (+ 1088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:15.892912 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:15.893893 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:15.893894 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:15.893957 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:15.893961 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 19:01:15.893992 (+ 31us) client_negotiation.cc:240] Negotiation successful
0929 19:01:15.894029 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":77,"threads_started":1}
I20260929 19:01:15.894469 7771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:15.889713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49186 (local address 127.0.98.2:35351)
0929 19:01:15.890452 (+ 739us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:15.890456 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:15.890465 (+ 9us) server_negotiation.cc:407] Connection header received
0929 19:01:15.890500 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:15.890502 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:15.890517 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:15.890713 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:01:15.890765 (+ 52us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:15.891737 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:15.892980 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:15.893779 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:15.894249 (+ 470us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:15.894331 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 19:01:15.894356 (+ 25us) server_negotiation.cc:299] Negotiation successful
0929 19:01:15.894396 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":700,"thread_start_us":69,"threads_started":1}
I20260929 19:01:15.894676 7408 tablet_service.cc:1467] Processing CreateTablet for tablet 0b2218d904bb45e98320b97c6b7d0a87 (DEFAULT_TABLE table=test-table [id=6995aade3cc540c280df3dcd1ee3609b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:01:15.894980 7408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b2218d904bb45e98320b97c6b7d0a87. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:01:15.896790 7774 tablet_bootstrap.cc:492] T 0b2218d904bb45e98320b97c6b7d0a87 P 07b2d889f6074dcf998c9ef656c568ca: Bootstrap starting.
I20260929 19:01:15.897231 7544 tablet_service.cc:1467] Processing CreateTablet for tablet 0b2218d904bb45e98320b97c6b7d0a87 (DEFAULT_TABLE table=test-table [id=6995aade3cc540c280df3dcd1ee3609b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:01:15.897518 7544 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b2218d904bb45e98320b97c6b7d0a87. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:01:15.897517 7774 tablet_bootstrap.cc:654] T 0b2218d904bb45e98320b97c6b7d0a87 P 07b2d889f6074dcf998c9ef656c568ca: Neither blocks nor log segments found. Creating new log.
I20260929 19:01:15.897843 7775 tablet_bootstrap.cc:492] T 0b2218d904bb45e98320b97c6b7d0a87 P 137b431b75d748d2bbc3786ad9df1faa: Bootstrap starting.
I20260929 19:01:15.897863 7774 log.cc:824] T 0b2218d904bb45e98320b97c6b7d0a87 P 07b2d889f6074dcf998c9ef656c568ca: Log is configured to *not* fsync() on all Append() calls
I20260929 19:01:15.898401 7775 tablet_bootstrap.cc:654] T 0b2218d904bb45e98320b97c6b7d0a87 P 137b431b75d748d2bbc3786ad9df1faa: Neither blocks nor log segments found. Creating new log.
I20260929 19:01:15.898715 7774 tablet_bootstrap.cc:492] T 0b2218d904bb45e98320b97c6b7d0a87 P 07b2d889f6074dcf998c9ef656c568ca: No bootstrap required, opened a new log
I20260929 19:01:15.898696 7775 log.cc:824] T 0b2218d904bb45e98320b97c6b7d0a87 P 137b431b75d748d2bbc3786ad9df1faa: Log is configured to *not* fsync() on all Append() calls
I20260929 19:01:15.898794 7774 ts_tablet_manager.cc:1397] T 0b2218d904bb45e98320b97c6b7d0a87 P 07b2d889f6074dcf998c9ef656c568ca: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260929 19:01:15.899608 7775 tablet_bootstrap.cc:492] T 0b2218d904bb45e98320b97c6b7d0a87 P 137b431b75d748d2bbc3786ad9df1faa: No bootstrap required, opened a new log
I20260929 19:01:15.899673 7775 ts_tablet_manager.cc:1397] T 0b2218d904bb45e98320b97c6b7d0a87 P 137b431b75d748d2bbc3786ad9df1faa: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 19:01:15.900758 7778 tablet_bootstrap.cc:492] T 0b2218d904bb45e98320b97c6b7d0a87 P 84528791e3a442dda71a633e7a3b6140: Bootstrap starting.
I20260929 19:01:15.900724 7774 raft_consensus.cc:348] T 0b2218d904bb45e98320b97c6b7d0a87 P 07b2d889f6074dcf998c9ef656c568ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07b2d889f6074dcf998c9ef656c568ca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42589 } } peers { permanent_uuid: "84528791e3a442dda71a633e7a3b6140" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35351 } } peers { permanent_uuid: "137b431b75d748d2bbc3786ad9df1faa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46675 } }
I20260929 19:01:15.900911 7774 raft_consensus.cc:374] T 0b2218d904bb45e98320b97c6b7d0a87 P 07b2d889f6074dcf998c9ef656c568ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:01:15.900954 7774 raft_consensus.cc:729] T 0b2218d904bb45e98320b97c6b7d0a87 P 07b2d889f6074dcf998c9ef656c568ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07b2d889f6074dcf998c9ef656c568ca, State: Initialized, Role: FOLLOWER
I20260929 19:01:15.901078 7774 consensus_queue.cc:260] T 0b2218d904bb45e98320b97c6b7d0a87 P 07b2d889f6074dcf998c9ef656c568ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07b2d889f6074dcf998c9ef656c568ca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42589 } } peers { permanent_uuid: "84528791e3a442dda71a633e7a3b6140" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35351 } } peers { permanent_uuid: "137b431b75d748d2bbc3786ad9df1faa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46675 } }
I20260929 19:01:15.901364 7774 ts_tablet_manager.cc:1428] T 0b2218d904bb45e98320b97c6b7d0a87 P 07b2d889f6074dcf998c9ef656c568ca: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260929 19:01:15.901955 7778 tablet_bootstrap.cc:654] T 0b2218d904bb45e98320b97c6b7d0a87 P 84528791e3a442dda71a633e7a3b6140: Neither blocks nor log segments found. Creating new log.
I20260929 19:01:15.902022 7747 heartbeater.cc:503] Master 127.0.98.62:42245 was elected leader, sending a full tablet report...
I20260929 19:01:15.902073 7775 raft_consensus.cc:348] T 0b2218d904bb45e98320b97c6b7d0a87 P 137b431b75d748d2bbc3786ad9df1faa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07b2d889f6074dcf998c9ef656c568ca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42589 } } peers { permanent_uuid: "84528791e3a442dda71a633e7a3b6140" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35351 } } peers { permanent_uuid: "137b431b75d748d2bbc3786ad9df1faa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46675 } }
I20260929 19:01:15.902221 7775 raft_consensus.cc:374] T 0b2218d904bb45e98320b97c6b7d0a87 P 137b431b75d748d2bbc3786ad9df1faa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:01:15.902258 7775 raft_consensus.cc:729] T 0b2218d904bb45e98320b97c6b7d0a87 P 137b431b75d748d2bbc3786ad9df1faa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 137b431b75d748d2bbc3786ad9df1faa, State: Initialized, Role: FOLLOWER
I20260929 19:01:15.902393 7775 consensus_queue.cc:260] T 0b2218d904bb45e98320b97c6b7d0a87 P 137b431b75d748d2bbc3786ad9df1faa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07b2d889f6074dcf998c9ef656c568ca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42589 } } peers { permanent_uuid: "84528791e3a442dda71a633e7a3b6140" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35351 } } peers { permanent_uuid: "137b431b75d748d2bbc3786ad9df1faa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46675 } }
I20260929 19:01:15.902621 7775 ts_tablet_manager.cc:1428] T 0b2218d904bb45e98320b97c6b7d0a87 P 137b431b75d748d2bbc3786ad9df1faa: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 19:01:15.902767 7475 heartbeater.cc:503] Master 127.0.98.62:42245 was elected leader, sending a full tablet report...
I20260929 19:01:15.902989 7778 log.cc:824] T 0b2218d904bb45e98320b97c6b7d0a87 P 84528791e3a442dda71a633e7a3b6140: Log is configured to *not* fsync() on all Append() calls
I20260929 19:01:15.903827 7778 tablet_bootstrap.cc:492] T 0b2218d904bb45e98320b97c6b7d0a87 P 84528791e3a442dda71a633e7a3b6140: No bootstrap required, opened a new log
I20260929 19:01:15.903889 7778 ts_tablet_manager.cc:1397] T 0b2218d904bb45e98320b97c6b7d0a87 P 84528791e3a442dda71a633e7a3b6140: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260929 19:01:15.906152 7778 raft_consensus.cc:348] T 0b2218d904bb45e98320b97c6b7d0a87 P 84528791e3a442dda71a633e7a3b6140 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07b2d889f6074dcf998c9ef656c568ca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42589 } } peers { permanent_uuid: "84528791e3a442dda71a633e7a3b6140" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35351 } } peers { permanent_uuid: "137b431b75d748d2bbc3786ad9df1faa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46675 } }
I20260929 19:01:15.906335 7778 raft_consensus.cc:374] T 0b2218d904bb45e98320b97c6b7d0a87 P 84528791e3a442dda71a633e7a3b6140 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:01:15.906381 7778 raft_consensus.cc:729] T 0b2218d904bb45e98320b97c6b7d0a87 P 84528791e3a442dda71a633e7a3b6140 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84528791e3a442dda71a633e7a3b6140, State: Initialized, Role: FOLLOWER
I20260929 19:01:15.906486 7778 consensus_queue.cc:260] T 0b2218d904bb45e98320b97c6b7d0a87 P 84528791e3a442dda71a633e7a3b6140 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07b2d889f6074dcf998c9ef656c568ca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42589 } } peers { permanent_uuid: "84528791e3a442dda71a633e7a3b6140" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35351 } } peers { permanent_uuid: "137b431b75d748d2bbc3786ad9df1faa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46675 } }
I20260929 19:01:15.906699 7778 ts_tablet_manager.cc:1428] T 0b2218d904bb45e98320b97c6b7d0a87 P 84528791e3a442dda71a633e7a3b6140: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 19:01:15.906749 7611 heartbeater.cc:503] Master 127.0.98.62:42245 was elected leader, sending a full tablet report...
I20260929 19:01:15.916776 7761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:15.914710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34244 (local address 127.0.98.60:39419)
0929 19:01:15.914762 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:15.914763 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:15.914771 (+ 8us) server_negotiation.cc:407] Connection header received
0929 19:01:15.914815 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:15.914816 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:15.914831 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:15.914875 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:01:15.914969 (+ 94us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:15.915425 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:15.916203 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:15.916562 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:15.916615 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:15.916651 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0929 19:01:15.916708 (+ 57us) server_negotiation.cc:299] Negotiation successful
0929 19:01:15.916730 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260929 19:01:15.916780 7772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:15.914650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39419 (local address 127.0.0.1:34244)
0929 19:01:15.914704 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:15.914716 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 19:01:15.914766 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:01:15.914887 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:01:15.914887 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:01:15.914889 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:01:15.914917 (+ 28us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:15.914919 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:15.915481 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:15.915481 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:15.916138 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:15.916139 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:15.916627 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:15.916628 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:15.916666 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:15.916668 (+ 2us) client_negotiation.cc:769] Sending connection context
0929 19:01:15.916706 (+ 38us) client_negotiation.cc:240] Negotiation successful
0929 19:01:15.916735 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260929 19:01:15.917241 7303 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 0908a5f3d25543efb7c1f184cd5178f4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.9] Dedup: 1.7->[1.8-1.9]
I20260929 19:01:15.917917 7332 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0908a5f3d25543efb7c1f184cd5178f4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ea566150fd024ed9b9485efbb8ee914c. Latest consensus state: current_term: 1 leader_uuid: "ea566150fd024ed9b9485efbb8ee914c" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ea566150fd024ed9b9485efbb8ee914c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42245 } } peers { permanent_uuid: "f5eac47223df4bf997b0f24a92898e8f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42823 } attrs { promote: false } } peers { permanent_uuid: "0908a5f3d25543efb7c1f184cd5178f4" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39419 } attrs { promote: true } } }
I20260929 19:01:15.918018 7332 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0908a5f3d25543efb7c1f184cd5178f4 [sys.catalog]: This master's current role is: LEARNER
W20260929 19:01:15.977448 7613 tablet.cc:2369] T 0b2218d904bb45e98320b97c6b7d0a87 P 84528791e3a442dda71a633e7a3b6140: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 19:01:16.035118 7782 raft_consensus.cc:482] T 0b2218d904bb45e98320b97c6b7d0a87 P 84528791e3a442dda71a633e7a3b6140 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 19:01:16.035228 7782 raft_consensus.cc:504] T 0b2218d904bb45e98320b97c6b7d0a87 P 84528791e3a442dda71a633e7a3b6140 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07b2d889f6074dcf998c9ef656c568ca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42589 } } peers { permanent_uuid: "84528791e3a442dda71a633e7a3b6140" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35351 } } peers { permanent_uuid: "137b431b75d748d2bbc3786ad9df1faa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46675 } }
I20260929 19:01:16.035621 7782 leader_election.cc:288] T 0b2218d904bb45e98320b97c6b7d0a87 P 84528791e3a442dda71a633e7a3b6140 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 07b2d889f6074dcf998c9ef656c568ca (127.0.98.3:42589), 137b431b75d748d2bbc3786ad9df1faa (127.0.98.1:46675)
I20260929 19:01:16.038480 7769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:16.035843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:32895 (local address 127.0.98.3:42589)
0929 19:01:16.035898 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:16.035901 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:16.035995 (+ 94us) server_negotiation.cc:407] Connection header received
0929 19:01:16.036038 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:16.036039 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:16.036050 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:16.036094 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:01:16.036287 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:16.036743 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:16.037758 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:16.038131 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:16.038212 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:16.038222 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0929 19:01:16.038380 (+ 158us) server_negotiation.cc:299] Negotiation successful
0929 19:01:16.038419 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260929 19:01:16.038480 7614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:16.035796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42589 (local address 127.0.98.2:32895)
0929 19:01:16.035895 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:16.035907 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 19:01:16.035959 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:01:16.036106 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:01:16.036107 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:01:16.036109 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:01:16.036210 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:16.036211 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:16.036924 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:16.036925 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:16.037656 (+ 731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:16.037658 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:16.038211 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:16.038213 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:16.038289 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:16.038292 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 19:01:16.038365 (+ 73us) client_negotiation.cc:240] Negotiation successful
0929 19:01:16.038404 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":84}
I20260929 19:01:16.038730 7616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:16.035843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46675 (local address 127.0.98.2:39423)
0929 19:01:16.035929 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:16.035941 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 19:01:16.035990 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:01:16.036177 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:01:16.036178 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:01:16.036179 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:01:16.036274 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:16.036275 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:16.037011 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:16.037012 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:16.037798 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:16.037800 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:16.038409 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:16.038410 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:16.038516 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:16.038519 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 19:01:16.038576 (+ 57us) client_negotiation.cc:240] Negotiation successful
0929 19:01:16.038634 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260929 19:01:16.038803 7700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b2218d904bb45e98320b97c6b7d0a87" candidate_uuid: "84528791e3a442dda71a633e7a3b6140" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07b2d889f6074dcf998c9ef656c568ca" is_pre_election: true
I20260929 19:01:16.038972 7700 raft_consensus.cc:2454] T 0b2218d904bb45e98320b97c6b7d0a87 P 07b2d889f6074dcf998c9ef656c568ca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84528791e3a442dda71a633e7a3b6140 in term 0.
I20260929 19:01:16.039038 7773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:16.035919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39423 (local address 127.0.98.1:46675)
0929 19:01:16.035966 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:16.035968 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:16.035977 (+ 9us) server_negotiation.cc:407] Connection header received
0929 19:01:16.036066 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:16.036068 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:16.036081 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:16.036174 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:01:16.036346 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:16.036917 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:16.037892 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:16.038311 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:16.038408 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:16.038417 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0929 19:01:16.038896 (+ 479us) server_negotiation.cc:299] Negotiation successful
0929 19:01:16.038945 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260929 19:01:16.039191 7497 leader_election.cc:302] T 0b2218d904bb45e98320b97c6b7d0a87 P 84528791e3a442dda71a633e7a3b6140 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07b2d889f6074dcf998c9ef656c568ca, 84528791e3a442dda71a633e7a3b6140; no voters:
I20260929 19:01:16.039314 7428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b2218d904bb45e98320b97c6b7d0a87" candidate_uuid: "84528791e3a442dda71a633e7a3b6140" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "137b431b75d748d2bbc3786ad9df1faa" is_pre_election: true
I20260929 19:01:16.039343 7782 raft_consensus.cc:2781] T 0b2218d904bb45e98320b97c6b7d0a87 P 84528791e3a442dda71a633e7a3b6140 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 19:01:16.039414 7782 raft_consensus.cc:482] T 0b2218d904bb45e98320b97c6b7d0a87 P 84528791e3a442dda71a633e7a3b6140 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 19:01:16.039444 7782 raft_consensus.cc:3037] T 0b2218d904bb45e98320b97c6b7d0a87 P 84528791e3a442dda71a633e7a3b6140 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:01:16.039465 7428 raft_consensus.cc:2454] T 0b2218d904bb45e98320b97c6b7d0a87 P 137b431b75d748d2bbc3786ad9df1faa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84528791e3a442dda71a633e7a3b6140 in term 0.
I20260929 19:01:16.040221 7782 raft_consensus.cc:504] T 0b2218d904bb45e98320b97c6b7d0a87 P 84528791e3a442dda71a633e7a3b6140 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07b2d889f6074dcf998c9ef656c568ca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42589 } } peers { permanent_uuid: "84528791e3a442dda71a633e7a3b6140" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35351 } } peers { permanent_uuid: "137b431b75d748d2bbc3786ad9df1faa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46675 } }
I20260929 19:01:16.040383 7782 leader_election.cc:288] T 0b2218d904bb45e98320b97c6b7d0a87 P 84528791e3a442dda71a633e7a3b6140 [CANDIDATE]: Term 1 election: Requested vote from peers 07b2d889f6074dcf998c9ef656c568ca (127.0.98.3:42589), 137b431b75d748d2bbc3786ad9df1faa (127.0.98.1:46675)
I20260929 19:01:16.040529 7428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b2218d904bb45e98320b97c6b7d0a87" candidate_uuid: "84528791e3a442dda71a633e7a3b6140" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "137b431b75d748d2bbc3786ad9df1faa"
I20260929 19:01:16.040550 7700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b2218d904bb45e98320b97c6b7d0a87" candidate_uuid: "84528791e3a442dda71a633e7a3b6140" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07b2d889f6074dcf998c9ef656c568ca"
I20260929 19:01:16.040621 7428 raft_consensus.cc:3037] T 0b2218d904bb45e98320b97c6b7d0a87 P 137b431b75d748d2bbc3786ad9df1faa [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:01:16.040648 7700 raft_consensus.cc:3037] T 0b2218d904bb45e98320b97c6b7d0a87 P 07b2d889f6074dcf998c9ef656c568ca [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:01:16.041384 7428 raft_consensus.cc:2454] T 0b2218d904bb45e98320b97c6b7d0a87 P 137b431b75d748d2bbc3786ad9df1faa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84528791e3a442dda71a633e7a3b6140 in term 1.
I20260929 19:01:16.041478 7700 raft_consensus.cc:2454] T 0b2218d904bb45e98320b97c6b7d0a87 P 07b2d889f6074dcf998c9ef656c568ca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84528791e3a442dda71a633e7a3b6140 in term 1.
I20260929 19:01:16.041579 7498 leader_election.cc:302] T 0b2218d904bb45e98320b97c6b7d0a87 P 84528791e3a442dda71a633e7a3b6140 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 137b431b75d748d2bbc3786ad9df1faa, 84528791e3a442dda71a633e7a3b6140; no voters:
I20260929 19:01:16.041694 7782 raft_consensus.cc:2781] T 0b2218d904bb45e98320b97c6b7d0a87 P 84528791e3a442dda71a633e7a3b6140 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:01:16.041870 7782 raft_consensus.cc:686] T 0b2218d904bb45e98320b97c6b7d0a87 P 84528791e3a442dda71a633e7a3b6140 [term 1 LEADER]: Becoming Leader. State: Replica: 84528791e3a442dda71a633e7a3b6140, State: Running, Role: LEADER
I20260929 19:01:16.041978 7782 consensus_queue.cc:237] T 0b2218d904bb45e98320b97c6b7d0a87 P 84528791e3a442dda71a633e7a3b6140 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07b2d889f6074dcf998c9ef656c568ca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42589 } } peers { permanent_uuid: "84528791e3a442dda71a633e7a3b6140" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35351 } } peers { permanent_uuid: "137b431b75d748d2bbc3786ad9df1faa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46675 } }
I20260929 19:01:16.042798 6509 catalog_manager.cc:5495] T 0b2218d904bb45e98320b97c6b7d0a87 P 84528791e3a442dda71a633e7a3b6140 reported cstate change: term changed from 0 to 1, leader changed from <none> to 84528791e3a442dda71a633e7a3b6140 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "84528791e3a442dda71a633e7a3b6140" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07b2d889f6074dcf998c9ef656c568ca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42589 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84528791e3a442dda71a633e7a3b6140" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35351 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "137b431b75d748d2bbc3786ad9df1faa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46675 } health_report { overall_health: UNKNOWN } } }
I20260929 19:01:16.043473 7789 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea566150fd024ed9b9485efbb8ee914c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ea566150fd024ed9b9485efbb8ee914c" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ea566150fd024ed9b9485efbb8ee914c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42245 } } peers { permanent_uuid: "f5eac47223df4bf997b0f24a92898e8f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42823 } attrs { promote: false } } peers { permanent_uuid: "0908a5f3d25543efb7c1f184cd5178f4" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39419 } attrs { promote: true } } }
I20260929 19:01:16.043579 7789 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea566150fd024ed9b9485efbb8ee914c [sys.catalog]: This master's current role is: LEADER
I20260929 19:01:16.044219 7764 raft_consensus.cc:1050] T 00000000000000000000000000000000 P ea566150fd024ed9b9485efbb8ee914c: attempting to promote NON_VOTER 0908a5f3d25543efb7c1f184cd5178f4 to VOTER
I20260929 19:01:16.044340 7764 consensus_queue.cc:237] T 00000000000000000000000000000000 P ea566150fd024ed9b9485efbb8ee914c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "ea566150fd024ed9b9485efbb8ee914c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42245 } } peers { permanent_uuid: "f5eac47223df4bf997b0f24a92898e8f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42823 } attrs { promote: false } } peers { permanent_uuid: "0908a5f3d25543efb7c1f184cd5178f4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39419 } attrs { promote: false } }
I20260929 19:01:16.044847 7117 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f5eac47223df4bf997b0f24a92898e8f [term 1 FOLLOWER]: Refusing update from remote peer ea566150fd024ed9b9485efbb8ee914c: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20260929 19:01:16.044912 7303 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0908a5f3d25543efb7c1f184cd5178f4 [term 1 LEARNER]: Refusing update from remote peer ea566150fd024ed9b9485efbb8ee914c: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20260929 19:01:16.045096 7789 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ea566150fd024ed9b9485efbb8ee914c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5eac47223df4bf997b0f24a92898e8f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42823 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260929 19:01:16.045185 7764 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ea566150fd024ed9b9485efbb8ee914c [LEADER]: Connected to new peer: Peer: permanent_uuid: "0908a5f3d25543efb7c1f184cd5178f4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39419 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260929 19:01:16.045993 7117 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f5eac47223df4bf997b0f24a92898e8f [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 7 to 11, 0908a5f3d25543efb7c1f184cd5178f4 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "ea566150fd024ed9b9485efbb8ee914c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42245 } } peers { permanent_uuid: "f5eac47223df4bf997b0f24a92898e8f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42823 } attrs { promote: false } } peers { permanent_uuid: "0908a5f3d25543efb7c1f184cd5178f4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39419 } attrs { promote: false } } }
I20260929 19:01:16.046025 7303 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 0908a5f3d25543efb7c1f184cd5178f4 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 7 to 11, 0908a5f3d25543efb7c1f184cd5178f4 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "ea566150fd024ed9b9485efbb8ee914c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42245 } } peers { permanent_uuid: "f5eac47223df4bf997b0f24a92898e8f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42823 } attrs { promote: false } } peers { permanent_uuid: "0908a5f3d25543efb7c1f184cd5178f4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39419 } attrs { promote: false } } }
I20260929 19:01:16.046826 7791 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5eac47223df4bf997b0f24a92898e8f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ea566150fd024ed9b9485efbb8ee914c" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "ea566150fd024ed9b9485efbb8ee914c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42245 } } peers { permanent_uuid: "f5eac47223df4bf997b0f24a92898e8f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42823 } attrs { promote: false } } peers { permanent_uuid: "0908a5f3d25543efb7c1f184cd5178f4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39419 } attrs { promote: false } } }
I20260929 19:01:16.045873 7789 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ea566150fd024ed9b9485efbb8ee914c [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 7 to 11, 0908a5f3d25543efb7c1f184cd5178f4 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "ea566150fd024ed9b9485efbb8ee914c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42245 } } peers { permanent_uuid: "f5eac47223df4bf997b0f24a92898e8f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42823 } attrs { promote: false } } peers { permanent_uuid: "0908a5f3d25543efb7c1f184cd5178f4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39419 } attrs { promote: false } } }
I20260929 19:01:16.046849 7332 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0908a5f3d25543efb7c1f184cd5178f4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ea566150fd024ed9b9485efbb8ee914c" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "ea566150fd024ed9b9485efbb8ee914c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42245 } } peers { permanent_uuid: "f5eac47223df4bf997b0f24a92898e8f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42823 } attrs { promote: false } } peers { permanent_uuid: "0908a5f3d25543efb7c1f184cd5178f4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39419 } attrs { promote: false } } }
I20260929 19:01:16.046922 7791 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5eac47223df4bf997b0f24a92898e8f [sys.catalog]: This master's current role is: FOLLOWER
I20260929 19:01:16.046926 7332 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0908a5f3d25543efb7c1f184cd5178f4 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 19:01:16.047765 7068 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea566150fd024ed9b9485efbb8ee914c [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "ea566150fd024ed9b9485efbb8ee914c" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "ea566150fd024ed9b9485efbb8ee914c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42245 } } peers { permanent_uuid: "f5eac47223df4bf997b0f24a92898e8f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42823 } attrs { promote: false } } peers { permanent_uuid: "0908a5f3d25543efb7c1f184cd5178f4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39419 } attrs { promote: false } } }
I20260929 19:01:16.047850 7789 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea566150fd024ed9b9485efbb8ee914c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ea566150fd024ed9b9485efbb8ee914c" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "ea566150fd024ed9b9485efbb8ee914c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42245 } } peers { permanent_uuid: "f5eac47223df4bf997b0f24a92898e8f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42823 } attrs { promote: false } } peers { permanent_uuid: "0908a5f3d25543efb7c1f184cd5178f4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39419 } attrs { promote: false } } }
I20260929 19:01:16.047890 7068 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea566150fd024ed9b9485efbb8ee914c [sys.catalog]: This master's current role is: LEADER
I20260929 19:01:16.047923 7789 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea566150fd024ed9b9485efbb8ee914c [sys.catalog]: This master's current role is: LEADER
I20260929 19:01:16.047765 7764 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea566150fd024ed9b9485efbb8ee914c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ea566150fd024ed9b9485efbb8ee914c" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "ea566150fd024ed9b9485efbb8ee914c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42245 } } peers { permanent_uuid: "f5eac47223df4bf997b0f24a92898e8f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42823 } attrs { promote: false } } peers { permanent_uuid: "0908a5f3d25543efb7c1f184cd5178f4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39419 } attrs { promote: false } } }
I20260929 19:01:16.047976 7764 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea566150fd024ed9b9485efbb8ee914c [sys.catalog]: This master's current role is: LEADER
I20260929 19:01:16.072615 7771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:16.070548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49188 (local address 127.0.98.2:35351)
0929 19:01:16.070586 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:16.070588 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:16.070595 (+ 7us) server_negotiation.cc:407] Connection header received
0929 19:01:16.070669 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:16.070669 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:16.070683 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:16.070732 (+ 49us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 19:01:16.070874 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:16.071305 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:16.071946 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:16.072106 (+ 160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:16.072188 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:16.072330 (+ 142us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 19:01:16.072417 (+ 87us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 19:01:16.072477 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0929 19:01:16.072510 (+ 33us) server_negotiation.cc:299] Negotiation successful
0929 19:01:16.072524 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260929 19:01:16.074213 7700 raft_consensus.cc:1261] T 0b2218d904bb45e98320b97c6b7d0a87 P 07b2d889f6074dcf998c9ef656c568ca [term 1 FOLLOWER]: Refusing update from remote peer 84528791e3a442dda71a633e7a3b6140: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 19:01:16.074218 7428 raft_consensus.cc:1261] T 0b2218d904bb45e98320b97c6b7d0a87 P 137b431b75d748d2bbc3786ad9df1faa [term 1 FOLLOWER]: Refusing update from remote peer 84528791e3a442dda71a633e7a3b6140: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 19:01:16.074522 7782 consensus_queue.cc:1035] T 0b2218d904bb45e98320b97c6b7d0a87 P 84528791e3a442dda71a633e7a3b6140 [LEADER]: Connected to new peer: Peer: permanent_uuid: "137b431b75d748d2bbc3786ad9df1faa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 19:01:16.074676 7787 consensus_queue.cc:1035] T 0b2218d904bb45e98320b97c6b7d0a87 P 84528791e3a442dda71a633e7a3b6140 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07b2d889f6074dcf998c9ef656c568ca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 19:01:16.077184 7794 mvcc.cc:205] Tried to move back new op lower bound from 7334741917996834816 to 7334741917867692032. Current Snapshot: MvccSnapshot[applied={T|T < 7334741917996834816}]
I20260929 19:01:16.078671 7796 mvcc.cc:205] Tried to move back new op lower bound from 7334741917996834816 to 7334741917867692032. Current Snapshot: MvccSnapshot[applied={T|T < 7334741917996834816}]
I20260929 19:01:16.080219 6510 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54636:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 19:01:16.080324 6510 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54636:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 19:01:16.081629 6510 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ea566150fd024ed9b9485efbb8ee914c: Sending DeleteTablet for 3 replicas of tablet 0b2218d904bb45e98320b97c6b7d0a87
I20260929 19:01:16.082319 7544 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b2218d904bb45e98320b97c6b7d0a87 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:01:16 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49186
I20260929 19:01:16.082469 7408 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b2218d904bb45e98320b97c6b7d0a87 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:01:16 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33890
I20260929 19:01:16.082535 7680 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b2218d904bb45e98320b97c6b7d0a87 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:01:16 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52252
I20260929 19:01:16.082687 7804 tablet_replica.cc:330] T 0b2218d904bb45e98320b97c6b7d0a87 P 84528791e3a442dda71a633e7a3b6140: stopping tablet replica
I20260929 19:01:16.082742 7806 tablet_replica.cc:330] T 0b2218d904bb45e98320b97c6b7d0a87 P 07b2d889f6074dcf998c9ef656c568ca: stopping tablet replica
I20260929 19:01:16.082787 7804 raft_consensus.cc:2229] T 0b2218d904bb45e98320b97c6b7d0a87 P 84528791e3a442dda71a633e7a3b6140 [term 1 LEADER]: Raft consensus shutting down.
I20260929 19:01:16.082912 7804 raft_consensus.cc:2258] T 0b2218d904bb45e98320b97c6b7d0a87 P 84528791e3a442dda71a633e7a3b6140 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 19:01:16.082952 7806 raft_consensus.cc:2229] T 0b2218d904bb45e98320b97c6b7d0a87 P 07b2d889f6074dcf998c9ef656c568ca [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 19:01:16.083132 7806 raft_consensus.cc:2258] T 0b2218d904bb45e98320b97c6b7d0a87 P 07b2d889f6074dcf998c9ef656c568ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 19:01:16.083290 7804 ts_tablet_manager.cc:1905] T 0b2218d904bb45e98320b97c6b7d0a87 P 84528791e3a442dda71a633e7a3b6140: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 19:01:16.083477 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUbnLHE/build/release/bin/kudu with pid 6478
I20260929 19:01:16.083526 7806 ts_tablet_manager.cc:1905] T 0b2218d904bb45e98320b97c6b7d0a87 P 07b2d889f6074dcf998c9ef656c568ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 19:01:16.085278 7806 ts_tablet_manager.cc:1918] T 0b2218d904bb45e98320b97c6b7d0a87 P 07b2d889f6074dcf998c9ef656c568ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 19:01:16.085359 7806 log.cc:1196] T 0b2218d904bb45e98320b97c6b7d0a87 P 07b2d889f6074dcf998c9ef656c568ca: Deleting WAL directory at /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/minicluster-data/ts-2/wal/wals/0b2218d904bb45e98320b97c6b7d0a87
I20260929 19:01:16.085602 7806 ts_tablet_manager.cc:1939] T 0b2218d904bb45e98320b97c6b7d0a87 P 07b2d889f6074dcf998c9ef656c568ca: Deleting consensus metadata
W20260929 19:01:16.089221 7635 connection.cc:535] server connection from 127.0.0.1:52252 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 19:01:16.090822 7804 ts_tablet_manager.cc:1918] T 0b2218d904bb45e98320b97c6b7d0a87 P 84528791e3a442dda71a633e7a3b6140: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 19:01:16.090893 7804 log.cc:1196] T 0b2218d904bb45e98320b97c6b7d0a87 P 84528791e3a442dda71a633e7a3b6140: Deleting WAL directory at /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/minicluster-data/ts-1/wal/wals/0b2218d904bb45e98320b97c6b7d0a87
I20260929 19:01:16.091153 7804 ts_tablet_manager.cc:1939] T 0b2218d904bb45e98320b97c6b7d0a87 P 84528791e3a442dda71a633e7a3b6140: Deleting consensus metadata
I20260929 19:01:16.091198 7805 tablet_replica.cc:330] T 0b2218d904bb45e98320b97c6b7d0a87 P 137b431b75d748d2bbc3786ad9df1faa: stopping tablet replica
I20260929 19:01:16.091298 7805 raft_consensus.cc:2229] T 0b2218d904bb45e98320b97c6b7d0a87 P 137b431b75d748d2bbc3786ad9df1faa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 19:01:16.091406 7805 raft_consensus.cc:2258] T 0b2218d904bb45e98320b97c6b7d0a87 P 137b431b75d748d2bbc3786ad9df1faa [term 1 FOLLOWER]: Raft consensus is shut down!
W20260929 19:01:16.091465 7497 connection.cc:441] server connection from 127.0.0.1:49186 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49186 (request call id 1) could send its response
I20260929 19:01:16.091861 7805 ts_tablet_manager.cc:1905] T 0b2218d904bb45e98320b97c6b7d0a87 P 137b431b75d748d2bbc3786ad9df1faa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 19:01:16.093029 7805 ts_tablet_manager.cc:1918] T 0b2218d904bb45e98320b97c6b7d0a87 P 137b431b75d748d2bbc3786ad9df1faa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 19:01:16.093099 7805 log.cc:1196] T 0b2218d904bb45e98320b97c6b7d0a87 P 137b431b75d748d2bbc3786ad9df1faa: Deleting WAL directory at /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/minicluster-data/ts-0/wal/wals/0b2218d904bb45e98320b97c6b7d0a87
I20260929 19:01:16.093349 7805 ts_tablet_manager.cc:1939] T 0b2218d904bb45e98320b97c6b7d0a87 P 137b431b75d748d2bbc3786ad9df1faa: Deleting consensus metadata
W20260929 19:01:16.093663 7361 connection.cc:441] server connection from 127.0.0.1:33890 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33890 (request call id 1) could send its response
I20260929 19:01:16.097847 7759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:16.090864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45008 (local address 127.0.98.61:42823)
0929 19:01:16.090913 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:16.090915 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:16.091061 (+ 146us) server_negotiation.cc:407] Connection header received
0929 19:01:16.091159 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:16.091160 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:16.091173 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:16.091203 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:01:16.091952 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:16.092666 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:16.093725 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:16.093961 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:16.094034 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:16.095286 (+ 1252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:01:16.095287 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:01:16.095288 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:01:16.095294 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:01:16.096629 (+ 1335us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:16.097055 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:16.097056 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:16.097056 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:16.097116 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:16.097311 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:16.097312 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:16.097312 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:16.097434 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:01:16.097532 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 19:01:16.097642 (+ 110us) server_negotiation.cc:299] Negotiation successful
0929 19:01:16.097721 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260929 19:01:16.104043 7761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:16.091067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34254 (local address 127.0.98.60:39419)
0929 19:01:16.091122 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:16.091124 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:16.097832 (+ 6708us) server_negotiation.cc:407] Connection header received
0929 19:01:16.097961 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:16.097962 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:16.097973 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:16.098020 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:01:16.098594 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:16.099375 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:16.099990 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:16.100321 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:16.100390 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:16.101510 (+ 1120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:01:16.101511 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:01:16.101512 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:01:16.101518 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:01:16.103052 (+ 1534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:16.103347 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:16.103348 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:16.103349 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:16.103408 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:16.103571 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:16.103572 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:16.103572 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:16.103695 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:01:16.103787 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0929 19:01:16.103863 (+ 76us) server_negotiation.cc:299] Negotiation successful
0929 19:01:16.103950 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260929 19:01:17.079051 7815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:17.078819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42245 (local address 127.0.98.2:32865)
0929 19:01:17.078922 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:17.078992 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87,"thread_start_us":44,"threads_started":1}
I20260929 19:01:17.079120 7813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:17.078699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42245 (local address 127.0.98.3:50461)
0929 19:01:17.078995 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:17.079070 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":65,"threads_started":1}
I20260929 19:01:17.079489 7814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:17.078767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42245 (local address 127.0.98.1:52541)
0929 19:01:17.079360 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:17.079440 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":578,"thread_start_us":52,"threads_started":1}
W20260929 19:01:17.079500 7747 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:42245 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:42245: connect: Connection refused (error 111)
W20260929 19:01:17.079586 7611 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:42245 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:42245: connect: Connection refused (error 111)
W20260929 19:01:17.079667 7475 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:42245 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:42245: connect: Connection refused (error 111)
I20260929 19:01:17.582020 7816 raft_consensus.cc:482] T 00000000000000000000000000000000 P f5eac47223df4bf997b0f24a92898e8f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ea566150fd024ed9b9485efbb8ee914c)
I20260929 19:01:17.582185 7817 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0908a5f3d25543efb7c1f184cd5178f4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ea566150fd024ed9b9485efbb8ee914c)
I20260929 19:01:17.582201 7816 raft_consensus.cc:504] T 00000000000000000000000000000000 P f5eac47223df4bf997b0f24a92898e8f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "ea566150fd024ed9b9485efbb8ee914c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42245 } } peers { permanent_uuid: "f5eac47223df4bf997b0f24a92898e8f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42823 } attrs { promote: false } } peers { permanent_uuid: "0908a5f3d25543efb7c1f184cd5178f4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39419 } attrs { promote: false } }
I20260929 19:01:17.582305 7817 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0908a5f3d25543efb7c1f184cd5178f4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "ea566150fd024ed9b9485efbb8ee914c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42245 } } peers { permanent_uuid: "f5eac47223df4bf997b0f24a92898e8f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42823 } attrs { promote: false } } peers { permanent_uuid: "0908a5f3d25543efb7c1f184cd5178f4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39419 } attrs { promote: false } }
I20260929 19:01:17.582455 7816 leader_election.cc:288] T 00000000000000000000000000000000 P f5eac47223df4bf997b0f24a92898e8f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ea566150fd024ed9b9485efbb8ee914c (127.0.98.62:42245), 0908a5f3d25543efb7c1f184cd5178f4 (127.0.98.60:39419)
I20260929 19:01:17.582702 7817 leader_election.cc:288] T 00000000000000000000000000000000 P 0908a5f3d25543efb7c1f184cd5178f4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ea566150fd024ed9b9485efbb8ee914c (127.0.98.62:42245), f5eac47223df4bf997b0f24a92898e8f (127.0.98.61:42823)
I20260929 19:01:17.582887 7818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:17.582586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42245 (local address 127.0.0.1:55134)
0929 19:01:17.582690 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:17.582843 (+ 153us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":51,"threads_started":1}
I20260929 19:01:17.582904 7820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:17.582586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42245 (local address 127.0.0.1:55118)
0929 19:01:17.582825 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:17.582865 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":48,"threads_started":1}
W20260929 19:01:17.583042 7091 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42245: connect: Connection refused (error 111)
W20260929 19:01:17.583073 7277 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42245: connect: Connection refused (error 111)
I20260929 19:01:17.583489 7818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:17.583404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42245 (local address 127.0.0.1:55146)
0929 19:01:17.583438 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:17.583460 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
I20260929 19:01:17.583544 7819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:17.583457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42245 (local address 127.0.0.1:55148)
0929 19:01:17.583492 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:17.583515 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260929 19:01:17.583611 7091 leader_election.cc:334] T 00000000000000000000000000000000 P f5eac47223df4bf997b0f24a92898e8f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ea566150fd024ed9b9485efbb8ee914c (127.0.98.62:42245): Network error: Client connection negotiation failed: client connection to 127.0.98.62:42245: connect: Connection refused (error 111)
W20260929 19:01:17.583714 7277 leader_election.cc:334] T 00000000000000000000000000000000 P 0908a5f3d25543efb7c1f184cd5178f4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ea566150fd024ed9b9485efbb8ee914c (127.0.98.62:42245): Network error: Client connection negotiation failed: client connection to 127.0.98.62:42245: connect: Connection refused (error 111)
I20260929 19:01:17.586452 7820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:17.582732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39419 (local address 127.0.0.1:34256)
0929 19:01:17.582952 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:17.582961 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 19:01:17.583037 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:01:17.583266 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:01:17.583267 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:01:17.583269 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:01:17.583326 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:17.583327 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:17.584288 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:17.584289 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:17.585468 (+ 1179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:17.585470 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:17.586194 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:17.586195 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:17.586257 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:17.586261 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 19:01:17.586316 (+ 55us) client_negotiation.cc:240] Negotiation successful
0929 19:01:17.586372 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212,"mutex_wait_us":2,"thread_start_us":49,"threads_started":1}
I20260929 19:01:17.586459 7822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:17.582878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34256 (local address 127.0.98.60:39419)
0929 19:01:17.582958 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:17.582960 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:17.583010 (+ 50us) server_negotiation.cc:407] Connection header received
0929 19:01:17.583132 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:17.583134 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:17.583146 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:17.583194 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:01:17.583402 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:17.584187 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:17.585560 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:17.586103 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:17.586181 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:17.586193 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0929 19:01:17.586333 (+ 140us) server_negotiation.cc:299] Negotiation successful
0929 19:01:17.586377 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":65,"thread_start_us":44,"threads_started":1}
I20260929 19:01:17.586460 7823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:17.582939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45018 (local address 127.0.98.61:42823)
0929 19:01:17.583014 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:17.583015 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:17.583180 (+ 165us) server_negotiation.cc:407] Connection header received
0929 19:01:17.583272 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:17.583273 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:17.583281 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:17.583322 (+ 41us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:01:17.583486 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:17.584212 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:17.585557 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:17.586103 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:17.586181 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:17.586193 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0929 19:01:17.586333 (+ 140us) server_negotiation.cc:299] Negotiation successful
0929 19:01:17.586397 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":61,"thread_start_us":42,"threads_started":1}
I20260929 19:01:17.586580 7821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:17.582819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42823 (local address 127.0.0.1:45018)
0929 19:01:17.583115 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:17.583135 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 19:01:17.583204 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:01:17.583372 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:01:17.583372 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:01:17.583375 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:01:17.583417 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:17.583418 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:17.584294 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:17.584294 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:17.585468 (+ 1174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:17.585470 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:17.586194 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:17.586195 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:17.586256 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:17.586260 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 19:01:17.586316 (+ 56us) client_negotiation.cc:240] Negotiation successful
0929 19:01:17.586361 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":53,"threads_started":1}
I20260929 19:01:17.586696 7303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f5eac47223df4bf997b0f24a92898e8f" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "0908a5f3d25543efb7c1f184cd5178f4" is_pre_election: true
I20260929 19:01:17.586784 7303 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0908a5f3d25543efb7c1f184cd5178f4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f5eac47223df4bf997b0f24a92898e8f in term 1.
I20260929 19:01:17.586926 7091 leader_election.cc:302] T 00000000000000000000000000000000 P f5eac47223df4bf997b0f24a92898e8f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0908a5f3d25543efb7c1f184cd5178f4, f5eac47223df4bf997b0f24a92898e8f; no voters: ea566150fd024ed9b9485efbb8ee914c
I20260929 19:01:17.586951 7117 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0908a5f3d25543efb7c1f184cd5178f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "f5eac47223df4bf997b0f24a92898e8f" is_pre_election: true
I20260929 19:01:17.587031 7816 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f5eac47223df4bf997b0f24a92898e8f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260929 19:01:17.587086 7117 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f5eac47223df4bf997b0f24a92898e8f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0908a5f3d25543efb7c1f184cd5178f4 in term 1.
I20260929 19:01:17.587126 7816 raft_consensus.cc:482] T 00000000000000000000000000000000 P f5eac47223df4bf997b0f24a92898e8f [term 1 FOLLOWER]: Starting leader election (detected failure of leader ea566150fd024ed9b9485efbb8ee914c)
I20260929 19:01:17.587158 7816 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f5eac47223df4bf997b0f24a92898e8f [term 1 FOLLOWER]: Advancing to term 2
I20260929 19:01:17.587229 7277 leader_election.cc:302] T 00000000000000000000000000000000 P 0908a5f3d25543efb7c1f184cd5178f4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0908a5f3d25543efb7c1f184cd5178f4, f5eac47223df4bf997b0f24a92898e8f; no voters: ea566150fd024ed9b9485efbb8ee914c
I20260929 19:01:17.587330 7817 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0908a5f3d25543efb7c1f184cd5178f4 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260929 19:01:17.587363 7817 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0908a5f3d25543efb7c1f184cd5178f4 [term 1 FOLLOWER]: Starting leader election (detected failure of leader ea566150fd024ed9b9485efbb8ee914c)
I20260929 19:01:17.587381 7817 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0908a5f3d25543efb7c1f184cd5178f4 [term 1 FOLLOWER]: Advancing to term 2
I20260929 19:01:17.587997 7816 raft_consensus.cc:504] T 00000000000000000000000000000000 P f5eac47223df4bf997b0f24a92898e8f [term 2 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "ea566150fd024ed9b9485efbb8ee914c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42245 } } peers { permanent_uuid: "f5eac47223df4bf997b0f24a92898e8f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42823 } attrs { promote: false } } peers { permanent_uuid: "0908a5f3d25543efb7c1f184cd5178f4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39419 } attrs { promote: false } }
I20260929 19:01:17.588137 7816 leader_election.cc:288] T 00000000000000000000000000000000 P f5eac47223df4bf997b0f24a92898e8f [CANDIDATE]: Term 2 election: Requested vote from peers ea566150fd024ed9b9485efbb8ee914c (127.0.98.62:42245), 0908a5f3d25543efb7c1f184cd5178f4 (127.0.98.60:39419)
I20260929 19:01:17.588132 7817 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0908a5f3d25543efb7c1f184cd5178f4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "ea566150fd024ed9b9485efbb8ee914c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42245 } } peers { permanent_uuid: "f5eac47223df4bf997b0f24a92898e8f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42823 } attrs { promote: false } } peers { permanent_uuid: "0908a5f3d25543efb7c1f184cd5178f4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39419 } attrs { promote: false } }
I20260929 19:01:17.588248 7817 leader_election.cc:288] T 00000000000000000000000000000000 P 0908a5f3d25543efb7c1f184cd5178f4 [CANDIDATE]: Term 2 election: Requested vote from peers ea566150fd024ed9b9485efbb8ee914c (127.0.98.62:42245), f5eac47223df4bf997b0f24a92898e8f (127.0.98.61:42823)
I20260929 19:01:17.588407 7820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:17.588260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42245 (local address 127.0.0.1:55162)
0929 19:01:17.588350 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:17.588378 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260929 19:01:17.588397 7303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f5eac47223df4bf997b0f24a92898e8f" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "0908a5f3d25543efb7c1f184cd5178f4"
I20260929 19:01:17.588433 7821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:17.588350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42245 (local address 127.0.0.1:55168)
0929 19:01:17.588385 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:17.588413 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
I20260929 19:01:17.588469 7303 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 0908a5f3d25543efb7c1f184cd5178f4 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate f5eac47223df4bf997b0f24a92898e8f in current term 2: Already voted for candidate 0908a5f3d25543efb7c1f184cd5178f4 in this term.
I20260929 19:01:17.588475 7117 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0908a5f3d25543efb7c1f184cd5178f4" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "f5eac47223df4bf997b0f24a92898e8f"
I20260929 19:01:17.588548 7117 raft_consensus.cc:2379] T 00000000000000000000000000000000 P f5eac47223df4bf997b0f24a92898e8f [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 0908a5f3d25543efb7c1f184cd5178f4 in current term 2: Already voted for candidate f5eac47223df4bf997b0f24a92898e8f in this term.
I20260929 19:01:17.588836 7820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:17.588712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42245 (local address 127.0.0.1:55176)
0929 19:01:17.588763 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:17.588787 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260929 19:01:17.588894 7091 leader_election.cc:334] T 00000000000000000000000000000000 P f5eac47223df4bf997b0f24a92898e8f [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer ea566150fd024ed9b9485efbb8ee914c (127.0.98.62:42245): Network error: Client connection negotiation failed: client connection to 127.0.98.62:42245: connect: Connection refused (error 111)
I20260929 19:01:17.588929 7091 leader_election.cc:302] T 00000000000000000000000000000000 P f5eac47223df4bf997b0f24a92898e8f [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f5eac47223df4bf997b0f24a92898e8f; no voters: 0908a5f3d25543efb7c1f184cd5178f4, ea566150fd024ed9b9485efbb8ee914c
I20260929 19:01:17.588969 7821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:17.588887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42245 (local address 127.0.0.1:55186)
0929 19:01:17.588922 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:17.588943 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260929 19:01:17.589002 7816 raft_consensus.cc:2726] T 00000000000000000000000000000000 P f5eac47223df4bf997b0f24a92898e8f [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260929 19:01:17.589013 7277 leader_election.cc:334] T 00000000000000000000000000000000 P 0908a5f3d25543efb7c1f184cd5178f4 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer ea566150fd024ed9b9485efbb8ee914c (127.0.98.62:42245): Network error: Client connection negotiation failed: client connection to 127.0.98.62:42245: connect: Connection refused (error 111)
I20260929 19:01:17.589069 7277 leader_election.cc:302] T 00000000000000000000000000000000 P 0908a5f3d25543efb7c1f184cd5178f4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0908a5f3d25543efb7c1f184cd5178f4; no voters: ea566150fd024ed9b9485efbb8ee914c, f5eac47223df4bf997b0f24a92898e8f
I20260929 19:01:17.589129 7817 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 0908a5f3d25543efb7c1f184cd5178f4 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260929 19:01:18.080174 7827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:18.079968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42245 (local address 127.0.98.2:45989)
0929 19:01:18.080075 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:18.080117 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85,"thread_start_us":53,"threads_started":1}
I20260929 19:01:18.080287 7828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:18.079983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42245 (local address 127.0.98.1:36749)
0929 19:01:18.080201 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:18.080238 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":88,"threads_started":1}
I20260929 19:01:18.080363 7826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:18.079939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42245 (local address 127.0.98.3:39541)
0929 19:01:18.080292 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:18.080324 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":86,"threads_started":1}
I20260929 19:01:19.081135 7831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:19.080925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42245 (local address 127.0.98.1:48761)
0929 19:01:19.081039 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:19.081081 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":49,"threads_started":1}
I20260929 19:01:19.081135 7830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:19.080790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42245 (local address 127.0.98.3:54211)
0929 19:01:19.081015 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:19.081081 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":67,"threads_started":1}
I20260929 19:01:19.081431 7829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:19.080625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42245 (local address 127.0.98.2:44221)
0929 19:01:19.081199 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:19.081394 (+ 195us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":65,"threads_started":1}
I20260929 19:01:19.193399 7832 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0908a5f3d25543efb7c1f184cd5178f4 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 19:01:19.193527 7832 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0908a5f3d25543efb7c1f184cd5178f4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "ea566150fd024ed9b9485efbb8ee914c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42245 } } peers { permanent_uuid: "f5eac47223df4bf997b0f24a92898e8f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42823 } attrs { promote: false } } peers { permanent_uuid: "0908a5f3d25543efb7c1f184cd5178f4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39419 } attrs { promote: false } }
I20260929 19:01:19.193727 7832 leader_election.cc:288] T 00000000000000000000000000000000 P 0908a5f3d25543efb7c1f184cd5178f4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ea566150fd024ed9b9485efbb8ee914c (127.0.98.62:42245), f5eac47223df4bf997b0f24a92898e8f (127.0.98.61:42823)
I20260929 19:01:19.194038 7833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:19.193848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42245 (local address 127.0.0.1:43312)
0929 19:01:19.193957 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:19.194003 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":56,"threads_started":1}
I20260929 19:01:19.194139 7117 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0908a5f3d25543efb7c1f184cd5178f4" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "f5eac47223df4bf997b0f24a92898e8f" is_pre_election: true
I20260929 19:01:19.194247 7117 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f5eac47223df4bf997b0f24a92898e8f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0908a5f3d25543efb7c1f184cd5178f4 in term 2.
I20260929 19:01:19.194427 7277 leader_election.cc:302] T 00000000000000000000000000000000 P 0908a5f3d25543efb7c1f184cd5178f4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0908a5f3d25543efb7c1f184cd5178f4, f5eac47223df4bf997b0f24a92898e8f; no voters:
I20260929 19:01:19.194466 7833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:19.194389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42245 (local address 127.0.0.1:43316)
0929 19:01:19.194418 (+ 29us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:19.194445 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
W20260929 19:01:19.194514 7277 leader_election.cc:334] T 00000000000000000000000000000000 P 0908a5f3d25543efb7c1f184cd5178f4 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer ea566150fd024ed9b9485efbb8ee914c (127.0.98.62:42245): Network error: Client connection negotiation failed: client connection to 127.0.98.62:42245: connect: Connection refused (error 111)
I20260929 19:01:19.194547 7832 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0908a5f3d25543efb7c1f184cd5178f4 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260929 19:01:19.194589 7832 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0908a5f3d25543efb7c1f184cd5178f4 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 19:01:19.194619 7832 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0908a5f3d25543efb7c1f184cd5178f4 [term 2 FOLLOWER]: Advancing to term 3
I20260929 19:01:19.195313 7832 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0908a5f3d25543efb7c1f184cd5178f4 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "ea566150fd024ed9b9485efbb8ee914c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42245 } } peers { permanent_uuid: "f5eac47223df4bf997b0f24a92898e8f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42823 } attrs { promote: false } } peers { permanent_uuid: "0908a5f3d25543efb7c1f184cd5178f4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39419 } attrs { promote: false } }
I20260929 19:01:19.195443 7832 leader_election.cc:288] T 00000000000000000000000000000000 P 0908a5f3d25543efb7c1f184cd5178f4 [CANDIDATE]: Term 3 election: Requested vote from peers ea566150fd024ed9b9485efbb8ee914c (127.0.98.62:42245), f5eac47223df4bf997b0f24a92898e8f (127.0.98.61:42823)
I20260929 19:01:19.195626 7833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:19.195540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42245 (local address 127.0.0.1:43322)
0929 19:01:19.195587 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:19.195611 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260929 19:01:19.195703 7117 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0908a5f3d25543efb7c1f184cd5178f4" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "f5eac47223df4bf997b0f24a92898e8f"
I20260929 19:01:19.195768 7117 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f5eac47223df4bf997b0f24a92898e8f [term 2 FOLLOWER]: Advancing to term 3
I20260929 19:01:19.195998 7833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:19.195894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42245 (local address 127.0.0.1:43336)
0929 19:01:19.195924 (+ 30us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:19.195979 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
W20260929 19:01:19.196085 7277 leader_election.cc:334] T 00000000000000000000000000000000 P 0908a5f3d25543efb7c1f184cd5178f4 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer ea566150fd024ed9b9485efbb8ee914c (127.0.98.62:42245): Network error: Client connection negotiation failed: client connection to 127.0.98.62:42245: connect: Connection refused (error 111)
I20260929 19:01:19.196480 7117 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f5eac47223df4bf997b0f24a92898e8f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0908a5f3d25543efb7c1f184cd5178f4 in term 3.
I20260929 19:01:19.196651 7277 leader_election.cc:302] T 00000000000000000000000000000000 P 0908a5f3d25543efb7c1f184cd5178f4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0908a5f3d25543efb7c1f184cd5178f4, f5eac47223df4bf997b0f24a92898e8f; no voters: ea566150fd024ed9b9485efbb8ee914c
I20260929 19:01:19.196743 7832 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0908a5f3d25543efb7c1f184cd5178f4 [term 3 FOLLOWER]: Leader election won for term 3
I20260929 19:01:19.196866 7832 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0908a5f3d25543efb7c1f184cd5178f4 [term 3 LEADER]: Becoming Leader. State: Replica: 0908a5f3d25543efb7c1f184cd5178f4, State: Running, Role: LEADER
I20260929 19:01:19.196985 7832 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0908a5f3d25543efb7c1f184cd5178f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "ea566150fd024ed9b9485efbb8ee914c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42245 } } peers { permanent_uuid: "f5eac47223df4bf997b0f24a92898e8f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42823 } attrs { promote: false } } peers { permanent_uuid: "0908a5f3d25543efb7c1f184cd5178f4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39419 } attrs { promote: false } }
I20260929 19:01:19.197355 7835 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0908a5f3d25543efb7c1f184cd5178f4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0908a5f3d25543efb7c1f184cd5178f4. Latest consensus state: current_term: 3 leader_uuid: "0908a5f3d25543efb7c1f184cd5178f4" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "ea566150fd024ed9b9485efbb8ee914c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42245 } } peers { permanent_uuid: "f5eac47223df4bf997b0f24a92898e8f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42823 } attrs { promote: false } } peers { permanent_uuid: "0908a5f3d25543efb7c1f184cd5178f4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39419 } attrs { promote: false } } }
I20260929 19:01:19.197484 7835 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0908a5f3d25543efb7c1f184cd5178f4 [sys.catalog]: This master's current role is: LEADER
I20260929 19:01:19.197664 7837 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 19:01:19.198027 7837 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 19:01:19.198081 7837 catalog_manager.cc:1259] Loaded cluster ID: 273b8d19129f4d25a0e7c884d5c7d851
I20260929 19:01:19.198097 7837 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 19:01:19.198236 7837 catalog_manager.cc:1504] Loading token signing keys...
I20260929 19:01:19.198283 7837 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 0908a5f3d25543efb7c1f184cd5178f4: Loaded TSK: 0
I20260929 19:01:19.198379 7837 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 19:01:19.260226 7293 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34254:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 19:01:19.262307 7293 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 19:01:19.263451 7117 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f5eac47223df4bf997b0f24a92898e8f [term 3 FOLLOWER]: Refusing update from remote peer 0908a5f3d25543efb7c1f184cd5178f4: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260929 19:01:19.263482 7833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:19.263284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42245 (local address 127.0.0.1:43344)
0929 19:01:19.263424 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:19.263455 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
I20260929 19:01:19.263711 7832 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0908a5f3d25543efb7c1f184cd5178f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5eac47223df4bf997b0f24a92898e8f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42823 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260929 19:01:19.263830 7833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:19.263735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42245 (local address 127.0.0.1:43360)
0929 19:01:19.263781 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:19.263812 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260929 19:01:19.263965 7277 consensus_peers.cc:487] T 00000000000000000000000000000000 P 0908a5f3d25543efb7c1f184cd5178f4 -> Peer ea566150fd024ed9b9485efbb8ee914c (127.0.98.62:42245): Couldn't send request to peer ea566150fd024ed9b9485efbb8ee914c. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42245: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260929 19:01:19.264355 7839 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5eac47223df4bf997b0f24a92898e8f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0908a5f3d25543efb7c1f184cd5178f4. Latest consensus state: current_term: 3 leader_uuid: "0908a5f3d25543efb7c1f184cd5178f4" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "ea566150fd024ed9b9485efbb8ee914c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42245 } } peers { permanent_uuid: "f5eac47223df4bf997b0f24a92898e8f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42823 } attrs { promote: false } } peers { permanent_uuid: "0908a5f3d25543efb7c1f184cd5178f4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39419 } attrs { promote: false } } }
I20260929 19:01:19.264477 7839 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5eac47223df4bf997b0f24a92898e8f [sys.catalog]: This master's current role is: FOLLOWER
I20260929 19:01:19.264827 7838 mvcc.cc:205] Tried to move back new op lower bound from 7334741931059937280 to 7334741930791084032. Current Snapshot: MvccSnapshot[applied={T|T < 7334741931059937280}]
I20260929 19:01:19.264922 7835 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0908a5f3d25543efb7c1f184cd5178f4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "0908a5f3d25543efb7c1f184cd5178f4" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "ea566150fd024ed9b9485efbb8ee914c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42245 } } peers { permanent_uuid: "f5eac47223df4bf997b0f24a92898e8f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42823 } attrs { promote: false } } peers { permanent_uuid: "0908a5f3d25543efb7c1f184cd5178f4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39419 } attrs { promote: false } } }
I20260929 19:01:19.265017 7835 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0908a5f3d25543efb7c1f184cd5178f4 [sys.catalog]: This master's current role is: LEADER
I20260929 19:01:19.264977 7839 sys_catalog.cc:455] T 00000000000000000000000000000000 P f5eac47223df4bf997b0f24a92898e8f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "0908a5f3d25543efb7c1f184cd5178f4" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "ea566150fd024ed9b9485efbb8ee914c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42245 } } peers { permanent_uuid: "f5eac47223df4bf997b0f24a92898e8f" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42823 } attrs { promote: false } } peers { permanent_uuid: "0908a5f3d25543efb7c1f184cd5178f4" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39419 } attrs { promote: false } } }
I20260929 19:01:19.265049 7839 sys_catalog.cc:458] T 00000000000000000000000000000000 P f5eac47223df4bf997b0f24a92898e8f [sys.catalog]: This master's current role is: FOLLOWER
I20260929 19:01:19.265141 7840 mvcc.cc:205] Tried to move back new op lower bound from 7334741931059937280 to 7334741930791084032. Current Snapshot: MvccSnapshot[applied={T|T < 7334741931059937280}]
I20260929 19:01:19.270025 7846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:19.266811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35351 (local address 127.0.0.1:50250)
0929 19:01:19.266932 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:19.266941 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 19:01:19.266981 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:01:19.267291 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:01:19.267292 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:01:19.267294 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:01:19.267321 (+ 27us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:19.267322 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:19.268263 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:19.268264 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:19.269107 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:19.269109 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:19.269842 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:19.269843 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:19.269899 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:19.269902 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 19:01:19.269949 (+ 47us) client_negotiation.cc:240] Negotiation successful
0929 19:01:19.269978 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"mutex_wait_us":1,"thread_start_us":80,"threads_started":1}
I20260929 19:01:19.270334 7848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:19.267033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50250 (local address 127.0.98.2:35351)
0929 19:01:19.267134 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:19.267137 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:19.267145 (+ 8us) server_negotiation.cc:407] Connection header received
0929 19:01:19.267170 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:19.267171 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:19.267181 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:19.267249 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:01:19.267393 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:19.268181 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:19.269204 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:19.269766 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:19.269840 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:19.269858 (+ 18us) server_negotiation.cc:1035] Waiting for connection context
0929 19:01:19.270006 (+ 148us) server_negotiation.cc:299] Negotiation successful
0929 19:01:19.270255 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81,"thread_start_us":53,"threads_started":1}
I20260929 19:01:19.270519 7844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:19.266776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42589 (local address 127.0.0.1:39312)
0929 19:01:19.266896 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:19.266906 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 19:01:19.266973 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:01:19.267243 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:01:19.267244 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:01:19.267247 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:01:19.267311 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:19.267314 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:19.268249 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:19.268249 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:19.269370 (+ 1121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:19.269371 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:19.270275 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:19.270276 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:19.270340 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:19.270344 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 19:01:19.270395 (+ 51us) client_negotiation.cc:240] Negotiation successful
0929 19:01:19.270447 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"mutex_wait_us":36,"thread_start_us":66,"threads_started":1}
I20260929 19:01:19.270512 7847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:19.267006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39312 (local address 127.0.98.3:42589)
0929 19:01:19.267092 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:19.267094 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:19.267103 (+ 9us) server_negotiation.cc:407] Connection header received
0929 19:01:19.267129 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:19.267130 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:19.267141 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:19.267379 (+ 238us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:01:19.267426 (+ 47us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:19.268165 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:19.269446 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:19.270159 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:19.270271 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:19.270283 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0929 19:01:19.270402 (+ 119us) server_negotiation.cc:299] Negotiation successful
0929 19:01:19.270446 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67,"thread_start_us":46,"threads_started":1}
I20260929 19:01:19.270744 7544 tablet_service.cc:1467] Processing CreateTablet for tablet 79e67d72ebf942aca86a0981def03038 (DEFAULT_TABLE table=test-table [id=51c9cfdc7b5f44fba7741fed7c24a00c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:01:19.270851 7680 tablet_service.cc:1467] Processing CreateTablet for tablet 79e67d72ebf942aca86a0981def03038 (DEFAULT_TABLE table=test-table [id=51c9cfdc7b5f44fba7741fed7c24a00c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:01:19.270881 7544 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79e67d72ebf942aca86a0981def03038. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:01:19.270953 7680 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79e67d72ebf942aca86a0981def03038. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:01:19.271334 7833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:19.266709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46675 (local address 127.0.0.1:48798)
0929 19:01:19.266757 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:19.266767 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 19:01:19.266814 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:01:19.267318 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:01:19.267319 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:01:19.267320 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:01:19.267375 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:19.267377 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:19.268138 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:19.268139 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:19.269360 (+ 1221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:19.269362 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:19.271078 (+ 1716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:19.271079 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:19.271140 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:19.271143 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 19:01:19.271229 (+ 86us) client_negotiation.cc:240] Negotiation successful
0929 19:01:19.271268 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260929 19:01:19.271623 7845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:19.266797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48798 (local address 127.0.98.1:46675)
0929 19:01:19.267144 (+ 347us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:19.267146 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:19.267152 (+ 6us) server_negotiation.cc:407] Connection header received
0929 19:01:19.267210 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:19.267210 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:19.267221 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:19.267454 (+ 233us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:01:19.267491 (+ 37us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:19.268040 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:19.270318 (+ 2278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:19.270972 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:19.271074 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:19.271086 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0929 19:01:19.271237 (+ 151us) server_negotiation.cc:299] Negotiation successful
0929 19:01:19.271274 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":52,"threads_started":1}
I20260929 19:01:19.271932 7408 tablet_service.cc:1467] Processing CreateTablet for tablet 79e67d72ebf942aca86a0981def03038 (DEFAULT_TABLE table=test-table [id=51c9cfdc7b5f44fba7741fed7c24a00c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:01:19.272068 7408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79e67d72ebf942aca86a0981def03038. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:01:19.272279 7849 tablet_bootstrap.cc:492] T 79e67d72ebf942aca86a0981def03038 P 84528791e3a442dda71a633e7a3b6140: Bootstrap starting.
I20260929 19:01:19.272420 7850 tablet_bootstrap.cc:492] T 79e67d72ebf942aca86a0981def03038 P 07b2d889f6074dcf998c9ef656c568ca: Bootstrap starting.
I20260929 19:01:19.272780 7850 tablet_bootstrap.cc:654] T 79e67d72ebf942aca86a0981def03038 P 07b2d889f6074dcf998c9ef656c568ca: Neither blocks nor log segments found. Creating new log.
I20260929 19:01:19.273384 7849 tablet_bootstrap.cc:654] T 79e67d72ebf942aca86a0981def03038 P 84528791e3a442dda71a633e7a3b6140: Neither blocks nor log segments found. Creating new log.
I20260929 19:01:19.273396 7850 tablet_bootstrap.cc:492] T 79e67d72ebf942aca86a0981def03038 P 07b2d889f6074dcf998c9ef656c568ca: No bootstrap required, opened a new log
I20260929 19:01:19.273463 7850 ts_tablet_manager.cc:1397] T 79e67d72ebf942aca86a0981def03038 P 07b2d889f6074dcf998c9ef656c568ca: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260929 19:01:19.273595 7850 raft_consensus.cc:348] T 79e67d72ebf942aca86a0981def03038 P 07b2d889f6074dcf998c9ef656c568ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "137b431b75d748d2bbc3786ad9df1faa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46675 } } peers { permanent_uuid: "07b2d889f6074dcf998c9ef656c568ca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42589 } } peers { permanent_uuid: "84528791e3a442dda71a633e7a3b6140" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35351 } }
I20260929 19:01:19.273669 7850 raft_consensus.cc:374] T 79e67d72ebf942aca86a0981def03038 P 07b2d889f6074dcf998c9ef656c568ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:01:19.273697 7850 raft_consensus.cc:729] T 79e67d72ebf942aca86a0981def03038 P 07b2d889f6074dcf998c9ef656c568ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07b2d889f6074dcf998c9ef656c568ca, State: Initialized, Role: FOLLOWER
I20260929 19:01:19.273739 7850 consensus_queue.cc:260] T 79e67d72ebf942aca86a0981def03038 P 07b2d889f6074dcf998c9ef656c568ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "137b431b75d748d2bbc3786ad9df1faa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46675 } } peers { permanent_uuid: "07b2d889f6074dcf998c9ef656c568ca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42589 } } peers { permanent_uuid: "84528791e3a442dda71a633e7a3b6140" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35351 } }
I20260929 19:01:19.273779 7852 tablet_bootstrap.cc:492] T 79e67d72ebf942aca86a0981def03038 P 137b431b75d748d2bbc3786ad9df1faa: Bootstrap starting.
I20260929 19:01:19.273928 7850 ts_tablet_manager.cc:1428] T 79e67d72ebf942aca86a0981def03038 P 07b2d889f6074dcf998c9ef656c568ca: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260929 19:01:19.274164 7852 tablet_bootstrap.cc:654] T 79e67d72ebf942aca86a0981def03038 P 137b431b75d748d2bbc3786ad9df1faa: Neither blocks nor log segments found. Creating new log.
I20260929 19:01:19.274308 7830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:19.274205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42245 (local address 127.0.98.3:50691)
0929 19:01:19.274249 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:19.274283 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
I20260929 19:01:19.274669 7852 tablet_bootstrap.cc:492] T 79e67d72ebf942aca86a0981def03038 P 137b431b75d748d2bbc3786ad9df1faa: No bootstrap required, opened a new log
I20260929 19:01:19.274739 7852 ts_tablet_manager.cc:1397] T 79e67d72ebf942aca86a0981def03038 P 137b431b75d748d2bbc3786ad9df1faa: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260929 19:01:19.274748 7849 tablet_bootstrap.cc:492] T 79e67d72ebf942aca86a0981def03038 P 84528791e3a442dda71a633e7a3b6140: No bootstrap required, opened a new log
I20260929 19:01:19.274794 7849 ts_tablet_manager.cc:1397] T 79e67d72ebf942aca86a0981def03038 P 84528791e3a442dda71a633e7a3b6140: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260929 19:01:19.274894 7852 raft_consensus.cc:348] T 79e67d72ebf942aca86a0981def03038 P 137b431b75d748d2bbc3786ad9df1faa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "137b431b75d748d2bbc3786ad9df1faa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46675 } } peers { permanent_uuid: "07b2d889f6074dcf998c9ef656c568ca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42589 } } peers { permanent_uuid: "84528791e3a442dda71a633e7a3b6140" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35351 } }
I20260929 19:01:19.274932 7849 raft_consensus.cc:348] T 79e67d72ebf942aca86a0981def03038 P 84528791e3a442dda71a633e7a3b6140 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "137b431b75d748d2bbc3786ad9df1faa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46675 } } peers { permanent_uuid: "07b2d889f6074dcf998c9ef656c568ca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42589 } } peers { permanent_uuid: "84528791e3a442dda71a633e7a3b6140" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35351 } }
I20260929 19:01:19.274979 7852 raft_consensus.cc:374] T 79e67d72ebf942aca86a0981def03038 P 137b431b75d748d2bbc3786ad9df1faa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:01:19.275020 7852 raft_consensus.cc:729] T 79e67d72ebf942aca86a0981def03038 P 137b431b75d748d2bbc3786ad9df1faa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 137b431b75d748d2bbc3786ad9df1faa, State: Initialized, Role: FOLLOWER
I20260929 19:01:19.275023 7849 raft_consensus.cc:374] T 79e67d72ebf942aca86a0981def03038 P 84528791e3a442dda71a633e7a3b6140 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:01:19.275048 7849 raft_consensus.cc:729] T 79e67d72ebf942aca86a0981def03038 P 84528791e3a442dda71a633e7a3b6140 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84528791e3a442dda71a633e7a3b6140, State: Initialized, Role: FOLLOWER
I20260929 19:01:19.275087 7746 heartbeater.cc:503] Master 127.0.98.60:39419 was elected leader, sending a full tablet report...
I20260929 19:01:19.275092 7849 consensus_queue.cc:260] T 79e67d72ebf942aca86a0981def03038 P 84528791e3a442dda71a633e7a3b6140 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "137b431b75d748d2bbc3786ad9df1faa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46675 } } peers { permanent_uuid: "07b2d889f6074dcf998c9ef656c568ca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42589 } } peers { permanent_uuid: "84528791e3a442dda71a633e7a3b6140" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35351 } }
I20260929 19:01:19.275089 7852 consensus_queue.cc:260] T 79e67d72ebf942aca86a0981def03038 P 137b431b75d748d2bbc3786ad9df1faa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "137b431b75d748d2bbc3786ad9df1faa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46675 } } peers { permanent_uuid: "07b2d889f6074dcf998c9ef656c568ca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42589 } } peers { permanent_uuid: "84528791e3a442dda71a633e7a3b6140" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35351 } }
I20260929 19:01:19.275264 7849 ts_tablet_manager.cc:1428] T 79e67d72ebf942aca86a0981def03038 P 84528791e3a442dda71a633e7a3b6140: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260929 19:01:19.275287 7852 ts_tablet_manager.cc:1428] T 79e67d72ebf942aca86a0981def03038 P 137b431b75d748d2bbc3786ad9df1faa: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 19:01:19.275627 7829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:19.275532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42245 (local address 127.0.98.2:45343)
0929 19:01:19.275567 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:19.275595 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
I20260929 19:01:19.275872 7474 heartbeater.cc:503] Master 127.0.98.60:39419 was elected leader, sending a full tablet report...
I20260929 19:01:19.275916 7831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:19.275768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42245 (local address 127.0.98.1:37461)
0929 19:01:19.275863 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:19.275888 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260929 19:01:19.276047 7610 heartbeater.cc:503] Master 127.0.98.60:39419 was elected leader, sending a full tablet report...
I20260929 19:01:19.276248 7831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:19.276159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42245 (local address 127.0.98.1:50523)
0929 19:01:19.276192 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:19.276218 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
I20260929 19:01:19.276435 7829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:19.276269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42245 (local address 127.0.98.2:37229)
0929 19:01:19.276385 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:19.276412 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
W20260929 19:01:19.352983 7478 tablet.cc:2369] T 79e67d72ebf942aca86a0981def03038 P 137b431b75d748d2bbc3786ad9df1faa: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 19:01:19.367712 7749 tablet.cc:2369] T 79e67d72ebf942aca86a0981def03038 P 07b2d889f6074dcf998c9ef656c568ca: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 19:01:19.448920 7856 raft_consensus.cc:482] T 79e67d72ebf942aca86a0981def03038 P 84528791e3a442dda71a633e7a3b6140 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 19:01:19.449013 7856 raft_consensus.cc:504] T 79e67d72ebf942aca86a0981def03038 P 84528791e3a442dda71a633e7a3b6140 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "137b431b75d748d2bbc3786ad9df1faa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46675 } } peers { permanent_uuid: "07b2d889f6074dcf998c9ef656c568ca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42589 } } peers { permanent_uuid: "84528791e3a442dda71a633e7a3b6140" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35351 } }
I20260929 19:01:19.449174 7856 leader_election.cc:288] T 79e67d72ebf942aca86a0981def03038 P 84528791e3a442dda71a633e7a3b6140 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 137b431b75d748d2bbc3786ad9df1faa (127.0.98.1:46675), 07b2d889f6074dcf998c9ef656c568ca (127.0.98.3:42589)
I20260929 19:01:19.449393 7428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79e67d72ebf942aca86a0981def03038" candidate_uuid: "84528791e3a442dda71a633e7a3b6140" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "137b431b75d748d2bbc3786ad9df1faa" is_pre_election: true
I20260929 19:01:19.449478 7428 raft_consensus.cc:2454] T 79e67d72ebf942aca86a0981def03038 P 137b431b75d748d2bbc3786ad9df1faa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84528791e3a442dda71a633e7a3b6140 in term 0.
I20260929 19:01:19.449401 7700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79e67d72ebf942aca86a0981def03038" candidate_uuid: "84528791e3a442dda71a633e7a3b6140" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07b2d889f6074dcf998c9ef656c568ca" is_pre_election: true
I20260929 19:01:19.449518 7700 raft_consensus.cc:2454] T 79e67d72ebf942aca86a0981def03038 P 07b2d889f6074dcf998c9ef656c568ca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84528791e3a442dda71a633e7a3b6140 in term 0.
I20260929 19:01:19.449642 7498 leader_election.cc:302] T 79e67d72ebf942aca86a0981def03038 P 84528791e3a442dda71a633e7a3b6140 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 137b431b75d748d2bbc3786ad9df1faa, 84528791e3a442dda71a633e7a3b6140; no voters:
I20260929 19:01:19.449748 7856 raft_consensus.cc:2781] T 79e67d72ebf942aca86a0981def03038 P 84528791e3a442dda71a633e7a3b6140 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 19:01:19.449816 7856 raft_consensus.cc:482] T 79e67d72ebf942aca86a0981def03038 P 84528791e3a442dda71a633e7a3b6140 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 19:01:19.449842 7856 raft_consensus.cc:3037] T 79e67d72ebf942aca86a0981def03038 P 84528791e3a442dda71a633e7a3b6140 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:01:19.450560 7856 raft_consensus.cc:504] T 79e67d72ebf942aca86a0981def03038 P 84528791e3a442dda71a633e7a3b6140 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "137b431b75d748d2bbc3786ad9df1faa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46675 } } peers { permanent_uuid: "07b2d889f6074dcf998c9ef656c568ca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42589 } } peers { permanent_uuid: "84528791e3a442dda71a633e7a3b6140" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35351 } }
I20260929 19:01:19.450716 7856 leader_election.cc:288] T 79e67d72ebf942aca86a0981def03038 P 84528791e3a442dda71a633e7a3b6140 [CANDIDATE]: Term 1 election: Requested vote from peers 137b431b75d748d2bbc3786ad9df1faa (127.0.98.1:46675), 07b2d889f6074dcf998c9ef656c568ca (127.0.98.3:42589)
I20260929 19:01:19.450860 7700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79e67d72ebf942aca86a0981def03038" candidate_uuid: "84528791e3a442dda71a633e7a3b6140" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07b2d889f6074dcf998c9ef656c568ca"
I20260929 19:01:19.450937 7700 raft_consensus.cc:3037] T 79e67d72ebf942aca86a0981def03038 P 07b2d889f6074dcf998c9ef656c568ca [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:01:19.450909 7428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79e67d72ebf942aca86a0981def03038" candidate_uuid: "84528791e3a442dda71a633e7a3b6140" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "137b431b75d748d2bbc3786ad9df1faa"
I20260929 19:01:19.451004 7428 raft_consensus.cc:3037] T 79e67d72ebf942aca86a0981def03038 P 137b431b75d748d2bbc3786ad9df1faa [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:01:19.451532 7700 raft_consensus.cc:2454] T 79e67d72ebf942aca86a0981def03038 P 07b2d889f6074dcf998c9ef656c568ca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84528791e3a442dda71a633e7a3b6140 in term 1.
I20260929 19:01:19.451624 7428 raft_consensus.cc:2454] T 79e67d72ebf942aca86a0981def03038 P 137b431b75d748d2bbc3786ad9df1faa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84528791e3a442dda71a633e7a3b6140 in term 1.
I20260929 19:01:19.451776 7497 leader_election.cc:302] T 79e67d72ebf942aca86a0981def03038 P 84528791e3a442dda71a633e7a3b6140 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07b2d889f6074dcf998c9ef656c568ca, 84528791e3a442dda71a633e7a3b6140; no voters:
I20260929 19:01:19.451906 7856 raft_consensus.cc:2781] T 79e67d72ebf942aca86a0981def03038 P 84528791e3a442dda71a633e7a3b6140 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:01:19.452057 7856 raft_consensus.cc:686] T 79e67d72ebf942aca86a0981def03038 P 84528791e3a442dda71a633e7a3b6140 [term 1 LEADER]: Becoming Leader. State: Replica: 84528791e3a442dda71a633e7a3b6140, State: Running, Role: LEADER
I20260929 19:01:19.452127 7856 consensus_queue.cc:237] T 79e67d72ebf942aca86a0981def03038 P 84528791e3a442dda71a633e7a3b6140 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "137b431b75d748d2bbc3786ad9df1faa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46675 } } peers { permanent_uuid: "07b2d889f6074dcf998c9ef656c568ca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42589 } } peers { permanent_uuid: "84528791e3a442dda71a633e7a3b6140" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35351 } }
I20260929 19:01:19.452376 7829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:19.452275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42245 (local address 127.0.98.2:39821)
0929 19:01:19.452317 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:19.452346 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260929 19:01:19.452889 7291 catalog_manager.cc:5495] T 79e67d72ebf942aca86a0981def03038 P 84528791e3a442dda71a633e7a3b6140 reported cstate change: term changed from 0 to 1, leader changed from <none> to 84528791e3a442dda71a633e7a3b6140 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "84528791e3a442dda71a633e7a3b6140" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "137b431b75d748d2bbc3786ad9df1faa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "07b2d889f6074dcf998c9ef656c568ca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42589 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84528791e3a442dda71a633e7a3b6140" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35351 } health_report { overall_health: HEALTHY } } }
I20260929 19:01:19.495860 7848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:19.493846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50254 (local address 127.0.98.2:35351)
0929 19:01:19.493911 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:19.493913 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:19.493922 (+ 9us) server_negotiation.cc:407] Connection header received
0929 19:01:19.493945 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:19.493946 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:19.493965 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:19.494012 (+ 47us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 19:01:19.494136 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:19.494600 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:19.495276 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:19.495452 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:19.495499 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:19.495609 (+ 110us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 19:01:19.495633 (+ 24us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 19:01:19.495670 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0929 19:01:19.495732 (+ 62us) server_negotiation.cc:299] Negotiation successful
0929 19:01:19.495749 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 19:01:19.496728 7428 raft_consensus.cc:1261] T 79e67d72ebf942aca86a0981def03038 P 137b431b75d748d2bbc3786ad9df1faa [term 1 FOLLOWER]: Refusing update from remote peer 84528791e3a442dda71a633e7a3b6140: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 19:01:19.496729 7700 raft_consensus.cc:1261] T 79e67d72ebf942aca86a0981def03038 P 07b2d889f6074dcf998c9ef656c568ca [term 1 FOLLOWER]: Refusing update from remote peer 84528791e3a442dda71a633e7a3b6140: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 19:01:19.496982 7856 consensus_queue.cc:1035] T 79e67d72ebf942aca86a0981def03038 P 84528791e3a442dda71a633e7a3b6140 [LEADER]: Connected to new peer: Peer: permanent_uuid: "137b431b75d748d2bbc3786ad9df1faa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 19:01:19.497100 7858 consensus_queue.cc:1035] T 79e67d72ebf942aca86a0981def03038 P 84528791e3a442dda71a633e7a3b6140 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07b2d889f6074dcf998c9ef656c568ca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42589 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 19:01:19.498029 7830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:19.497885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42245 (local address 127.0.98.3:35469)
0929 19:01:19.497954 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:19.497994 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260929 19:01:19.498325 7831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:19.498215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42245 (local address 127.0.98.1:36749)
0929 19:01:19.498252 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:19.498293 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
I20260929 19:01:19.498867 7829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:19.498774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42245 (local address 127.0.98.2:54727)
0929 19:01:19.498807 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:19.498837 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
I20260929 19:01:19.499006 7831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:19.498893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42245 (local address 127.0.98.1:48815)
0929 19:01:19.498957 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:19.498981 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260929 19:01:19.500229 7829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:19.500118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42245 (local address 127.0.98.2:46681)
0929 19:01:19.500178 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:19.500203 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260929 19:01:19.500586 7830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:19.500462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42245 (local address 127.0.98.3:59893)
0929 19:01:19.500501 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:19.500554 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260929 19:01:19.500680 7291 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34254:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 19:01:19.500770 7291 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34254:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 19:01:19.501996 7291 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0908a5f3d25543efb7c1f184cd5178f4: Sending DeleteTablet for 3 replicas of tablet 79e67d72ebf942aca86a0981def03038
I20260929 19:01:19.502350 7544 tablet_service.cc:1514] Processing DeleteTablet for tablet 79e67d72ebf942aca86a0981def03038 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:01:19 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:50250
I20260929 19:01:19.502389 7408 tablet_service.cc:1514] Processing DeleteTablet for tablet 79e67d72ebf942aca86a0981def03038 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:01:19 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:48798
I20260929 19:01:19.502539 7869 tablet_replica.cc:330] T 79e67d72ebf942aca86a0981def03038 P 84528791e3a442dda71a633e7a3b6140: stopping tablet replica
I20260929 19:01:19.502595 7869 raft_consensus.cc:2229] T 79e67d72ebf942aca86a0981def03038 P 84528791e3a442dda71a633e7a3b6140 [term 1 LEADER]: Raft consensus shutting down.
I20260929 19:01:19.502650 7869 raft_consensus.cc:2258] T 79e67d72ebf942aca86a0981def03038 P 84528791e3a442dda71a633e7a3b6140 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 19:01:19.502640 7680 tablet_service.cc:1514] Processing DeleteTablet for tablet 79e67d72ebf942aca86a0981def03038 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:01:19 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:39312
I20260929 19:01:19.502835 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUbnLHE/build/release/bin/kudu with pid 7345
I20260929 19:01:19.503706 7869 ts_tablet_manager.cc:1905] T 79e67d72ebf942aca86a0981def03038 P 84528791e3a442dda71a633e7a3b6140: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 19:01:19.505475 7871 tablet_replica.cc:330] T 79e67d72ebf942aca86a0981def03038 P 07b2d889f6074dcf998c9ef656c568ca: stopping tablet replica
I20260929 19:01:19.505569 7871 raft_consensus.cc:2229] T 79e67d72ebf942aca86a0981def03038 P 07b2d889f6074dcf998c9ef656c568ca [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 19:01:19.505622 7871 raft_consensus.cc:2258] T 79e67d72ebf942aca86a0981def03038 P 07b2d889f6074dcf998c9ef656c568ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 19:01:19.505685 7869 ts_tablet_manager.cc:1918] T 79e67d72ebf942aca86a0981def03038 P 84528791e3a442dda71a633e7a3b6140: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 19:01:19.505750 7869 log.cc:1196] T 79e67d72ebf942aca86a0981def03038 P 84528791e3a442dda71a633e7a3b6140: Deleting WAL directory at /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/minicluster-data/ts-1/wal/wals/79e67d72ebf942aca86a0981def03038
I20260929 19:01:19.506019 7869 ts_tablet_manager.cc:1939] T 79e67d72ebf942aca86a0981def03038 P 84528791e3a442dda71a633e7a3b6140: Deleting consensus metadata
I20260929 19:01:19.506343 7278 catalog_manager.cc:4841] TS 84528791e3a442dda71a633e7a3b6140 (127.0.98.2:35351): tablet 79e67d72ebf942aca86a0981def03038 (table test-table [id=51c9cfdc7b5f44fba7741fed7c24a00c]) successfully deleted
W20260929 19:01:19.509008 7833 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:01:19.508883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46675 (local address 127.0.0.1:48814)
0929 19:01:19.508923 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:19.508954 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46675: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":21}
I20260929 19:01:19.509032 7871 ts_tablet_manager.cc:1905] T 79e67d72ebf942aca86a0981def03038 P 07b2d889f6074dcf998c9ef656c568ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 19:01:19.509155 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUbnLHE/build/release/bin/kudu with pid 7481
W20260929 19:01:19.509162 7278 catalog_manager.cc:4584] TS 137b431b75d748d2bbc3786ad9df1faa (127.0.98.1:46675): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 79e67d72ebf942aca86a0981def03038: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46675: connect: Connection reset by peer (error 104)
I20260929 19:01:19.510689 7871 ts_tablet_manager.cc:1918] T 79e67d72ebf942aca86a0981def03038 P 07b2d889f6074dcf998c9ef656c568ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 19:01:19.510766 7871 log.cc:1196] T 79e67d72ebf942aca86a0981def03038 P 07b2d889f6074dcf998c9ef656c568ca: Deleting WAL directory at /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790708447806542-392-0/minicluster-data/ts-2/wal/wals/79e67d72ebf942aca86a0981def03038
I20260929 19:01:19.510974 7871 ts_tablet_manager.cc:1939] T 79e67d72ebf942aca86a0981def03038 P 07b2d889f6074dcf998c9ef656c568ca: Deleting consensus metadata
I20260929 19:01:19.511277 7277 catalog_manager.cc:4841] TS 07b2d889f6074dcf998c9ef656c568ca (127.0.98.3:42589): tablet 79e67d72ebf942aca86a0981def03038 (table test-table [id=51c9cfdc7b5f44fba7741fed7c24a00c]) successfully deleted
I20260929 19:01:19.516031 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUbnLHE/build/release/bin/kudu with pid 7617
I20260929 19:01:19.520681 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUbnLHE/build/release/bin/kudu with pid 6974
I20260929 19:01:19.525408 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUbnLHE/build/release/bin/kudu with pid 7161
2026-09-29T19:01:19Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (6392 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708447806542-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 19:01:19 dist-test-slave-lqkv krb5kdc[7874](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 19:01:19 dist-test-slave-lqkv krb5kdc[7874](info): set up 2 sockets
Sep 29 19:01:19 dist-test-slave-lqkv krb5kdc[7874](info): commencing operation
krb5kdc: starting...
W20260929 19:01:20.557281 392 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708447806542-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708447806542-392-0/krb5kdc/test-user.keytab.
Sep 29 19:01:20 dist-test-slave-lqkv krb5kdc[7874](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708480, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T19:01:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T19:01:20Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708447806542-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708447806542-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260929 19:01:20.671496 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUbnLHE/build/release/bin/kudu
/tmp/dist-test-taskUbnLHE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708447806542-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708447806542-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708447806542-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708447806542-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34221
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708447806542-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34221
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708447806542-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708447806542-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708447806542-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:01:20.750339 7887 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:01:20.750546 7887 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:01:20.750568 7887 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:01:20.751765 7887 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 19:01:20.751816 7887 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260929 19:01:20.751832 7887 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:01:20.751844 7887 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:01:20.751857 7887 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 19:01:20.751869 7887 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 19:01:20.753509 7887 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708447806542-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708447806542-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34221
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708447806542-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34221
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708447806542-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7887
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708447806542-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 18:09:13 UTC on ccycloud.quasar-hpkalf.root.comops.site
build id 4680913
I20260929 19:01:20.753716 7887 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:01:20.753892 7887 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:01:20.758505 7893 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:01:20.758602 7887 server_base.cc:1034] running on GCE node
W20260929 19:01:20.758512 7892 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:01:20.758518 7895 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:01:20.758903 7887 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:01:20.759176 7887 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:01:20.760316 7887 hybrid_clock.cc:648] HybridClock initialized: now 1790708480760297 us; error 37 us; skew 500 ppm
Sep 29 19:01:20 dist-test-slave-lqkv krb5kdc[7874](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708480, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:01:20.762712 7887 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260929 19:01:20.763161 7887 webserver.cc:460] Webserver started at http://127.0.98.62:33185/ using document root <none> and password file <none>
I20260929 19:01:20.763371 7887 fs_manager.cc:359] Metadata directory not provided
I20260929 19:01:20.763415 7887 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:01:20.763554 7887 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:01:20.764551 7887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708447806542-392-0/minicluster-data/master-0/data/instance:
uuid: "1db5b3fcae274bbf94cb5d9e6eff70fa"
format_stamp: "Formatted at 2026-09-29 19:01:20 on dist-test-slave-lqkv"
server_key: "418218b05a9aed180476c8b3f4aee291"
server_key_iv: "5c713f131ec652cecd74acde4349185b"
server_key_version: "encryptionkey@0"
I20260929 19:01:20.764935 7887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708447806542-392-0/minicluster-data/master-0/wal/instance:
uuid: "1db5b3fcae274bbf94cb5d9e6eff70fa"
format_stamp: "Formatted at 2026-09-29 19:01:20 on dist-test-slave-lqkv"
server_key: "418218b05a9aed180476c8b3f4aee291"
server_key_iv: "5c713f131ec652cecd74acde4349185b"
server_key_version: "encryptionkey@0"
I20260929 19:01:20.766189 7887 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 19:01:20.766871 7902 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:20.767051 7887 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 19:01:20.767138 7887 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708447806542-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708447806542-392-0/minicluster-data/master-0/wal
uuid: "1db5b3fcae274bbf94cb5d9e6eff70fa"
format_stamp: "Formatted at 2026-09-29 19:01:20 on dist-test-slave-lqkv"
server_key: "418218b05a9aed180476c8b3f4aee291"
server_key_iv: "5c713f131ec652cecd74acde4349185b"
server_key_version: "encryptionkey@0"
I20260929 19:01:20.767208 7887 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708447806542-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708447806542-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708447806542-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:01:20.796926 7887 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:01:20.798894 7887 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:01:20.799047 7887 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:01:20.803660 7887 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34221
I20260929 19:01:20.803706 7964 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34221 every 8 connection(s)
I20260929 19:01:20.804019 7887 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708447806542-392-0/minicluster-data/master-0/data/info.pb
I20260929 19:01:20.804610 7965 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:01:20.806046 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUbnLHE/build/release/bin/kudu as pid 7887
I20260929 19:01:20.806137 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708447806542-392-0/minicluster-data/master-0/wal/instance
I20260929 19:01:20.806360 392 external_mini_cluster.cc:1428] Setting key 6ba8329a70b0c7322e5ce299de84c8bb
I20260929 19:01:20.807047 7965 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1db5b3fcae274bbf94cb5d9e6eff70fa: Bootstrap starting.
I20260929 19:01:20.807606 7965 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1db5b3fcae274bbf94cb5d9e6eff70fa: Neither blocks nor log segments found. Creating new log.
I20260929 19:01:20.807983 7965 log.cc:824] T 00000000000000000000000000000000 P 1db5b3fcae274bbf94cb5d9e6eff70fa: Log is configured to *not* fsync() on all Append() calls
I20260929 19:01:20.808700 7965 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1db5b3fcae274bbf94cb5d9e6eff70fa: No bootstrap required, opened a new log
I20260929 19:01:20.810922 7965 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1db5b3fcae274bbf94cb5d9e6eff70fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1db5b3fcae274bbf94cb5d9e6eff70fa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34221 } }
I20260929 19:01:20.811059 7965 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1db5b3fcae274bbf94cb5d9e6eff70fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:01:20.811082 7965 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1db5b3fcae274bbf94cb5d9e6eff70fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1db5b3fcae274bbf94cb5d9e6eff70fa, State: Initialized, Role: FOLLOWER
I20260929 19:01:20.811250 7965 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1db5b3fcae274bbf94cb5d9e6eff70fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1db5b3fcae274bbf94cb5d9e6eff70fa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34221 } }
I20260929 19:01:20.811343 7965 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1db5b3fcae274bbf94cb5d9e6eff70fa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 19:01:20.811389 7965 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1db5b3fcae274bbf94cb5d9e6eff70fa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 19:01:20.811710 7965 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1db5b3fcae274bbf94cb5d9e6eff70fa [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:01:20.812234 7965 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1db5b3fcae274bbf94cb5d9e6eff70fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1db5b3fcae274bbf94cb5d9e6eff70fa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34221 } }
I20260929 19:01:20.812358 7965 leader_election.cc:302] T 00000000000000000000000000000000 P 1db5b3fcae274bbf94cb5d9e6eff70fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1db5b3fcae274bbf94cb5d9e6eff70fa; no voters:
I20260929 19:01:20.812531 7965 leader_election.cc:288] T 00000000000000000000000000000000 P 1db5b3fcae274bbf94cb5d9e6eff70fa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 19:01:20.812557 7970 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1db5b3fcae274bbf94cb5d9e6eff70fa [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:01:20.812678 7970 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1db5b3fcae274bbf94cb5d9e6eff70fa [term 1 LEADER]: Becoming Leader. State: Replica: 1db5b3fcae274bbf94cb5d9e6eff70fa, State: Running, Role: LEADER
I20260929 19:01:20.812786 7965 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1db5b3fcae274bbf94cb5d9e6eff70fa [sys.catalog]: configured and running, proceeding with master startup.
I20260929 19:01:20.812848 7970 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1db5b3fcae274bbf94cb5d9e6eff70fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1db5b3fcae274bbf94cb5d9e6eff70fa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34221 } }
I20260929 19:01:20.813171 7970 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1db5b3fcae274bbf94cb5d9e6eff70fa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1db5b3fcae274bbf94cb5d9e6eff70fa. Latest consensus state: current_term: 1 leader_uuid: "1db5b3fcae274bbf94cb5d9e6eff70fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1db5b3fcae274bbf94cb5d9e6eff70fa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34221 } } }
I20260929 19:01:20.813290 7972 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1db5b3fcae274bbf94cb5d9e6eff70fa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1db5b3fcae274bbf94cb5d9e6eff70fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1db5b3fcae274bbf94cb5d9e6eff70fa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34221 } } }
Sep 29 19:01:20 dist-test-slave-lqkv krb5kdc[7874](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708480, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260929 19:01:20.813319 7970 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1db5b3fcae274bbf94cb5d9e6eff70fa [sys.catalog]: This master's current role is: LEADER
I20260929 19:01:20.813380 7972 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1db5b3fcae274bbf94cb5d9e6eff70fa [sys.catalog]: This master's current role is: LEADER
I20260929 19:01:20.813649 7976 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 19:01:20.814196 7976 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 19:01:20.816062 7976 catalog_manager.cc:1347] Generated new cluster ID: 8246c763524942b0b29934234e48d3fd
I20260929 19:01:20.816145 7976 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 19:01:20.817619 7968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:20.806879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38766 (local address 127.0.98.62:34221)
0929 19:01:20.807053 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:20.807062 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:20.807094 (+ 32us) server_negotiation.cc:407] Connection header received
0929 19:01:20.807567 (+ 473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:20.807574 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:20.807620 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:20.807811 (+ 191us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:01:20.808579 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:20.809556 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:20.810677 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:20.811087 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:20.811191 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:20.813970 (+ 2779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:01:20.813974 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:01:20.813975 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:01:20.814028 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:01:20.816036 (+ 2008us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:20.816426 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:20.816429 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:20.816429 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:20.816523 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:20.816743 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:20.816744 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:20.816744 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:20.817035 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:01:20.817167 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0929 19:01:20.817309 (+ 142us) server_negotiation.cc:299] Negotiation successful
0929 19:01:20.817427 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":77,"threads_started":1}
I20260929 19:01:20.830755 7976 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 19:01:20.831436 7976 catalog_manager.cc:1504] Loading token signing keys...
I20260929 19:01:20.834970 7976 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1db5b3fcae274bbf94cb5d9e6eff70fa: Generated new TSK 0
I20260929 19:01:20.835126 7976 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708447806542-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708447806542-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260929 19:01:20.862843 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUbnLHE/build/release/bin/kudu
/tmp/dist-test-taskUbnLHE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708447806542-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708447806542-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708447806542-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708447806542-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34221
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708447806542-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708447806542-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708447806542-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708447806542-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:01:20.946201 7991 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:01:20.946386 7991 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:01:20.946411 7991 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:01:20.947618 7991 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:01:20.947656 7991 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:01:20.947681 7991 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 19:01:20.947700 7991 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 19:01:20.949285 7991 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708447806542-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708447806542-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708447806542-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708447806542-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:34221
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7991
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708447806542-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 18:09:13 UTC on ccycloud.quasar-hpkalf.root.comops.site
build id 4680913
I20260929 19:01:20.949508 7991 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:01:20.949781 7991 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:01:20.955474 7997 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:01:20.955591 7996 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:01:20.955709 7991 server_base.cc:1034] running on GCE node
W20260929 19:01:20.955470 7999 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:01:20.956043 7991 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:01:20.956302 7991 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:01:20.957451 7991 hybrid_clock.cc:648] HybridClock initialized: now 1790708480957432 us; error 35 us; skew 500 ppm
Sep 29 19:01:20 dist-test-slave-lqkv krb5kdc[7874](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708480, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:01:20.959880 7991 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260929 19:01:20.960248 7991 webserver.cc:460] Webserver started at http://127.0.98.1:33945/ using document root <none> and password file <none>
I20260929 19:01:20.960443 7991 fs_manager.cc:359] Metadata directory not provided
I20260929 19:01:20.960484 7991 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:01:20.960568 7991 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:01:20.961614 7991 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708447806542-392-0/minicluster-data/ts-0/data/instance:
uuid: "936984d765f745738bea4acbbe9c61fb"
format_stamp: "Formatted at 2026-09-29 19:01:20 on dist-test-slave-lqkv"
server_key: "338917fabd26e361299b1673af12631e"
server_key_iv: "ff1815c57ee7026cbe2149545e464865"
server_key_version: "encryptionkey@0"
I20260929 19:01:20.961905 7991 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708447806542-392-0/minicluster-data/ts-0/wal/instance:
uuid: "936984d765f745738bea4acbbe9c61fb"
format_stamp: "Formatted at 2026-09-29 19:01:20 on dist-test-slave-lqkv"
server_key: "338917fabd26e361299b1673af12631e"
server_key_iv: "ff1815c57ee7026cbe2149545e464865"
server_key_version: "encryptionkey@0"
I20260929 19:01:20.963094 7991 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260929 19:01:20.963748 8006 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:20.963924 7991 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 19:01:20.963994 7991 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708447806542-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708447806542-392-0/minicluster-data/ts-0/wal
uuid: "936984d765f745738bea4acbbe9c61fb"
format_stamp: "Formatted at 2026-09-29 19:01:20 on dist-test-slave-lqkv"
server_key: "338917fabd26e361299b1673af12631e"
server_key_iv: "ff1815c57ee7026cbe2149545e464865"
server_key_version: "encryptionkey@0"
I20260929 19:01:20.964097 7991 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708447806542-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708447806542-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708447806542-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:01:20.988785 7991 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:01:20.990674 7991 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:01:20.990862 7991 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:01:20.991362 7991 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:01:20.991429 7991 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:20.991473 7991 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:01:20.991516 7991 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 19:01:20 dist-test-slave-lqkv krb5kdc[7874](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708480, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260929 19:01:20.998643 7991 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37753
I20260929 19:01:20.998724 8121 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37753 every 8 connection(s)
I20260929 19:01:20.999073 7991 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708447806542-392-0/minicluster-data/ts-0/data/info.pb
I20260929 19:01:21.001458 7968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:20.991910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39787 (local address 127.0.98.62:34221)
0929 19:01:20.991975 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:20.991978 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:20.992185 (+ 207us) server_negotiation.cc:407] Connection header received
0929 19:01:20.992729 (+ 544us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:20.992731 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:20.992744 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:20.992794 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:01:20.993533 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:20.994039 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:20.995052 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:20.995386 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:20.995461 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:20.998829 (+ 3368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:01:20.998831 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:01:20.998831 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:01:20.998839 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:01:21.000228 (+ 1389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:21.000610 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:21.000611 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:21.000611 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:21.000658 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:21.000914 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:21.000915 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:21.000915 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:21.001021 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:01:21.001071 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0929 19:01:21.001315 (+ 244us) server_negotiation.cc:299] Negotiation successful
0929 19:01:21.001396 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 19:01:21.001597 8015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:20.991893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34221 (local address 127.0.98.1:39787)
0929 19:01:20.992065 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:20.992098 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0929 19:01:20.992606 (+ 508us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:01:20.992901 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:01:20.992905 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:01:20.993224 (+ 319us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:01:20.993417 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:20.993421 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:20.994131 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:20.994132 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:20.994961 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:20.994963 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:20.995488 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:20.995489 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:20.995685 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:20.996220 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:01:20.996231 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:01:20.998731 (+ 2500us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:01:21.000303 (+ 1572us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:01:21.000308 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:01:21.000310 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:01:21.000537 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:01:21.000716 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:01:21.000716 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:01:21.000716 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:01:21.000857 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:01:21.001077 (+ 220us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:01:21.001081 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:01:21.001188 (+ 107us) client_negotiation.cc:769] Sending connection context
0929 19:01:21.001277 (+ 89us) client_negotiation.cc:240] Negotiation successful
0929 19:01:21.001390 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":86,"threads_started":1}
I20260929 19:01:21.002116 8122 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34221
I20260929 19:01:21.002190 8122 heartbeater.cc:464] Registering TS with master...
I20260929 19:01:21.002360 8122 heartbeater.cc:511] Master 127.0.98.62:34221 requested a full tablet report, sending...
I20260929 19:01:21.002926 7919 ts_manager.cc:194] Registered new tserver with Master: 936984d765f745738bea4acbbe9c61fb (127.0.98.1:37753)
I20260929 19:01:21.003588 7919 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:39787
I20260929 19:01:21.008107 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUbnLHE/build/release/bin/kudu as pid 7991
I20260929 19:01:21.008222 7968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:21.003152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33119 (local address 127.0.98.62:34221)
0929 19:01:21.003193 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:21.003195 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:21.003268 (+ 73us) server_negotiation.cc:407] Connection header received
0929 19:01:21.003337 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:21.003338 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:21.003347 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:21.003390 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:01:21.003716 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:21.004375 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:21.005124 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:21.005296 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:21.005361 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:21.006175 (+ 814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:01:21.006175 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:01:21.006175 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:01:21.006180 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:01:21.007278 (+ 1098us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:21.007557 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:21.007557 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:21.007558 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:21.007600 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:21.007741 (+ 141us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:21.007741 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:21.007741 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:21.007818 (+ 77us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:01:21.007866 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0929 19:01:21.008071 (+ 205us) server_negotiation.cc:299] Negotiation successful
0929 19:01:21.008126 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260929 19:01:21.008231 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708447806542-392-0/minicluster-data/ts-0/wal/instance
I20260929 19:01:21.008281 8128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:21.003097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34221 (local address 127.0.98.1:33119)
0929 19:01:21.003214 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:21.003224 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 19:01:21.003283 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:01:21.003401 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:01:21.003403 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:01:21.003575 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:01:21.003620 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:21.003621 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:21.004472 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:21.004473 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:21.005061 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:21.005063 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:21.005360 (+ 297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:21.005361 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:21.005406 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:21.005831 (+ 425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:01:21.005842 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:01:21.006106 (+ 264us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:01:21.007344 (+ 1238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:01:21.007345 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:01:21.007345 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:01:21.007498 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:01:21.007658 (+ 160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:01:21.007659 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:01:21.007659 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:01:21.007705 (+ 46us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:01:21.007926 (+ 221us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:01:21.007927 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:01:21.007972 (+ 45us) client_negotiation.cc:769] Sending connection context
0929 19:01:21.008057 (+ 85us) client_negotiation.cc:240] Negotiation successful
0929 19:01:21.008154 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":65,"threads_started":1}
I20260929 19:01:21.008468 392 external_mini_cluster.cc:1428] Setting key 19a33dd0970cc94b03b13c5985384934
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708447806542-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708447806542-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260929 19:01:21.031445 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUbnLHE/build/release/bin/kudu
/tmp/dist-test-taskUbnLHE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708447806542-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708447806542-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708447806542-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708447806542-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34221
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708447806542-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708447806542-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708447806542-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708447806542-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:01:21.112208 8131 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:01:21.112480 8131 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:01:21.112527 8131 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:01:21.113837 8131 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:01:21.113909 8131 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:01:21.113943 8131 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 19:01:21.113968 8131 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 19:01:21.115518 8131 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708447806542-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708447806542-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708447806542-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708447806542-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:34221
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8131
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708447806542-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 18:09:13 UTC on ccycloud.quasar-hpkalf.root.comops.site
build id 4680913
I20260929 19:01:21.115739 8131 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:01:21.115954 8131 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:01:21.120685 8137 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:01:21.120698 8136 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:01:21.120684 8139 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:01:21.120961 8131 server_base.cc:1034] running on GCE node
I20260929 19:01:21.121162 8131 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:01:21.121397 8131 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:01:21.122567 8131 hybrid_clock.cc:648] HybridClock initialized: now 1790708481122552 us; error 33 us; skew 500 ppm
Sep 29 19:01:21 dist-test-slave-lqkv krb5kdc[7874](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708481, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:01:21.125031 8131 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260929 19:01:21.125654 8131 webserver.cc:460] Webserver started at http://127.0.98.2:42715/ using document root <none> and password file <none>
I20260929 19:01:21.125852 8131 fs_manager.cc:359] Metadata directory not provided
I20260929 19:01:21.125895 8131 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:01:21.125998 8131 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:01:21.126932 8131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708447806542-392-0/minicluster-data/ts-1/data/instance:
uuid: "b27ba18ec1f24f4f9691a5e7de5ff564"
format_stamp: "Formatted at 2026-09-29 19:01:21 on dist-test-slave-lqkv"
server_key: "0ef903c027285dd5ecb5eaa5a6241d41"
server_key_iv: "9fcb94c81de52eb04395032a4deeacdc"
server_key_version: "encryptionkey@0"
I20260929 19:01:21.127234 8131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708447806542-392-0/minicluster-data/ts-1/wal/instance:
uuid: "b27ba18ec1f24f4f9691a5e7de5ff564"
format_stamp: "Formatted at 2026-09-29 19:01:21 on dist-test-slave-lqkv"
server_key: "0ef903c027285dd5ecb5eaa5a6241d41"
server_key_iv: "9fcb94c81de52eb04395032a4deeacdc"
server_key_version: "encryptionkey@0"
I20260929 19:01:21.128461 8131 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 19:01:21.129160 8146 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:21.129320 8131 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:21.129415 8131 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708447806542-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708447806542-392-0/minicluster-data/ts-1/wal
uuid: "b27ba18ec1f24f4f9691a5e7de5ff564"
format_stamp: "Formatted at 2026-09-29 19:01:21 on dist-test-slave-lqkv"
server_key: "0ef903c027285dd5ecb5eaa5a6241d41"
server_key_iv: "9fcb94c81de52eb04395032a4deeacdc"
server_key_version: "encryptionkey@0"
I20260929 19:01:21.129498 8131 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708447806542-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708447806542-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708447806542-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:01:21.145970 8131 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:01:21.147804 8131 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:01:21.147984 8131 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:01:21.148492 8131 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:01:21.148556 8131 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:21.148599 8131 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:01:21.148629 8131 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 19:01:21 dist-test-slave-lqkv krb5kdc[7874](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708481, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260929 19:01:21.155402 8131 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37627
I20260929 19:01:21.155488 8261 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37627 every 8 connection(s)
I20260929 19:01:21.155772 8131 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708447806542-392-0/minicluster-data/ts-1/data/info.pb
I20260929 19:01:21.158546 7968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:21.148999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35143 (local address 127.0.98.62:34221)
0929 19:01:21.149074 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:21.149077 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:21.149250 (+ 173us) server_negotiation.cc:407] Connection header received
0929 19:01:21.149701 (+ 451us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:21.149703 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:21.149716 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:21.149800 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:01:21.150381 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:21.151238 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:21.152836 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:21.153184 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:21.153268 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:21.155967 (+ 2699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:01:21.155968 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:01:21.155969 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:01:21.155976 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:01:21.157230 (+ 1254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:21.157610 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:21.157611 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:21.157611 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:21.157666 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:21.157918 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:21.157919 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:21.157919 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:21.158039 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:01:21.158106 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0929 19:01:21.158376 (+ 270us) server_negotiation.cc:299] Negotiation successful
0929 19:01:21.158457 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260929 19:01:21.158722 8155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:21.148968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34221 (local address 127.0.98.2:35143)
0929 19:01:21.149112 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:21.149150 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 19:01:21.149609 (+ 459us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:01:21.149855 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:01:21.149859 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:01:21.150097 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:01:21.150272 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:21.150276 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:21.151339 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:21.151340 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:21.152716 (+ 1376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:21.152718 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:21.153293 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:21.153294 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:21.153414 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:21.153935 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:01:21.153948 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:01:21.155844 (+ 1896us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:01:21.157310 (+ 1466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:01:21.157316 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:01:21.157318 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:01:21.157533 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:01:21.157735 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:01:21.157736 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:01:21.157736 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:01:21.157845 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:01:21.158110 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:01:21.158116 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:01:21.158249 (+ 133us) client_negotiation.cc:769] Sending connection context
0929 19:01:21.158355 (+ 106us) client_negotiation.cc:240] Negotiation successful
0929 19:01:21.158503 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":76,"threads_started":1}
I20260929 19:01:21.159178 8262 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34221
I20260929 19:01:21.159261 8262 heartbeater.cc:464] Registering TS with master...
I20260929 19:01:21.159452 8262 heartbeater.cc:511] Master 127.0.98.62:34221 requested a full tablet report, sending...
I20260929 19:01:21.160094 7918 ts_manager.cc:194] Registered new tserver with Master: b27ba18ec1f24f4f9691a5e7de5ff564 (127.0.98.2:37627)
I20260929 19:01:21.160521 7918 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:35143
I20260929 19:01:21.165742 7968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:21.160194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44717 (local address 127.0.98.62:34221)
0929 19:01:21.160242 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:21.160244 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:21.160788 (+ 544us) server_negotiation.cc:407] Connection header received
0929 19:01:21.160883 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:21.160883 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:21.160893 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:21.160955 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:01:21.161260 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:21.161712 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:21.162393 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:21.162600 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:21.162674 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:21.163484 (+ 810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:01:21.163486 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:01:21.163486 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:01:21.163492 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:01:21.164554 (+ 1062us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:21.164949 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:21.164950 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:21.164950 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:21.165006 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:21.165213 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:21.165214 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:21.165214 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:21.165351 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:01:21.165423 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 19:01:21.165582 (+ 159us) server_negotiation.cc:299] Negotiation successful
0929 19:01:21.165656 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260929 19:01:21.165762 8268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:21.160121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34221 (local address 127.0.98.2:44717)
0929 19:01:21.160715 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:21.160725 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 19:01:21.160799 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:01:21.160960 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:01:21.160961 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:01:21.161131 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:01:21.161180 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:21.161182 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:21.161801 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:21.161802 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:21.162312 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:21.162313 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:21.162672 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:21.162673 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:21.162740 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:21.163081 (+ 341us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:01:21.163085 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:01:21.163360 (+ 275us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:01:21.164649 (+ 1289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:01:21.164650 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:01:21.164651 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:01:21.164872 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:01:21.165072 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:01:21.165072 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:01:21.165073 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:01:21.165138 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:01:21.165437 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:01:21.165438 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:01:21.165476 (+ 38us) client_negotiation.cc:769] Sending connection context
0929 19:01:21.165555 (+ 79us) client_negotiation.cc:240] Negotiation successful
0929 19:01:21.165662 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":574,"thread_start_us":112,"threads_started":1}
I20260929 19:01:21.165999 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUbnLHE/build/release/bin/kudu as pid 8131
I20260929 19:01:21.166113 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708447806542-392-0/minicluster-data/ts-1/wal/instance
I20260929 19:01:21.166322 392 external_mini_cluster.cc:1428] Setting key 24d329ea0d0277ffc69fc08f8c0e376b
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708447806542-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708447806542-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260929 19:01:21.189924 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUbnLHE/build/release/bin/kudu
/tmp/dist-test-taskUbnLHE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708447806542-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708447806542-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708447806542-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708447806542-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34221
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708447806542-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708447806542-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708447806542-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708447806542-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:01:21.269019 8271 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:01:21.269199 8271 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:01:21.269227 8271 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:01:21.270323 8271 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:01:21.270357 8271 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:01:21.270377 8271 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 19:01:21.270396 8271 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 19:01:21.271862 8271 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708447806542-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708447806542-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708447806542-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708447806542-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:34221
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8271
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708447806542-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 18:09:13 UTC on ccycloud.quasar-hpkalf.root.comops.site
build id 4680913
I20260929 19:01:21.272042 8271 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:01:21.272194 8271 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:01:21.276777 8276 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:01:21.276916 8271 server_base.cc:1034] running on GCE node
W20260929 19:01:21.276752 8279 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:01:21.276763 8277 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:01:21.277205 8271 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:01:21.277450 8271 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:01:21.278602 8271 hybrid_clock.cc:648] HybridClock initialized: now 1790708481278591 us; error 29 us; skew 500 ppm
Sep 29 19:01:21 dist-test-slave-lqkv krb5kdc[7874](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708481, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:01:21.281069 8271 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260929 19:01:21.281538 8271 webserver.cc:460] Webserver started at http://127.0.98.3:33337/ using document root <none> and password file <none>
I20260929 19:01:21.281760 8271 fs_manager.cc:359] Metadata directory not provided
I20260929 19:01:21.281836 8271 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:01:21.281987 8271 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:01:21.282989 8271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708447806542-392-0/minicluster-data/ts-2/data/instance:
uuid: "649983f98f554789af84afcb77b8e38a"
format_stamp: "Formatted at 2026-09-29 19:01:21 on dist-test-slave-lqkv"
server_key: "f3c01931fdb3a47d4c839aa941e24580"
server_key_iv: "88381fe41cf61e15350eb5d30050450b"
server_key_version: "encryptionkey@0"
I20260929 19:01:21.283351 8271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708447806542-392-0/minicluster-data/ts-2/wal/instance:
uuid: "649983f98f554789af84afcb77b8e38a"
format_stamp: "Formatted at 2026-09-29 19:01:21 on dist-test-slave-lqkv"
server_key: "f3c01931fdb3a47d4c839aa941e24580"
server_key_iv: "88381fe41cf61e15350eb5d30050450b"
server_key_version: "encryptionkey@0"
I20260929 19:01:21.284682 8271 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 19:01:21.285434 8286 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:21.285597 8271 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 19:01:21.285696 8271 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708447806542-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708447806542-392-0/minicluster-data/ts-2/wal
uuid: "649983f98f554789af84afcb77b8e38a"
format_stamp: "Formatted at 2026-09-29 19:01:21 on dist-test-slave-lqkv"
server_key: "f3c01931fdb3a47d4c839aa941e24580"
server_key_iv: "88381fe41cf61e15350eb5d30050450b"
server_key_version: "encryptionkey@0"
I20260929 19:01:21.285775 8271 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708447806542-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708447806542-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708447806542-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:01:21.300738 8271 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:01:21.303052 8271 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:01:21.303208 8271 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:01:21.303797 8271 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:01:21.303855 8271 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:21.303895 8271 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:01:21.303922 8271 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 19:01:21 dist-test-slave-lqkv krb5kdc[7874](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708481, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260929 19:01:21.311383 8271 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46835
I20260929 19:01:21.311493 8401 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46835 every 8 connection(s)
I20260929 19:01:21.311825 8271 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708447806542-392-0/minicluster-data/ts-2/data/info.pb
I20260929 19:01:21.313690 7968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:21.304267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57579 (local address 127.0.98.62:34221)
0929 19:01:21.304322 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:21.304324 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:21.304503 (+ 179us) server_negotiation.cc:407] Connection header received
0929 19:01:21.305119 (+ 616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:21.305121 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:21.305133 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:21.305199 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:01:21.305970 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:21.306519 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:21.307500 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:21.307707 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:21.307794 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:21.310440 (+ 2646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:01:21.310441 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:01:21.310441 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:01:21.310449 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:01:21.312177 (+ 1728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:21.312608 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:21.312609 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:21.312609 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:21.312690 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:21.312977 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:21.312977 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:21.312978 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:21.313104 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:01:21.313177 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 19:01:21.313535 (+ 358us) server_negotiation.cc:299] Negotiation successful
0929 19:01:21.313623 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260929 19:01:21.313901 8295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:21.304244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34221 (local address 127.0.98.3:57579)
0929 19:01:21.304387 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:21.304420 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0929 19:01:21.305012 (+ 592us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:01:21.305269 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:01:21.305273 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:01:21.305593 (+ 320us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:01:21.305886 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:21.305892 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:21.306612 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:21.306614 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:21.307408 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:21.307410 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:21.307802 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:21.307803 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:21.307928 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:21.308402 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:01:21.308409 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:01:21.310346 (+ 1937us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:01:21.312248 (+ 1902us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:01:21.312251 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:01:21.312253 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:01:21.312543 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:01:21.312757 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:01:21.312757 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:01:21.312757 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:01:21.312917 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:01:21.313183 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:01:21.313187 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:01:21.313349 (+ 162us) client_negotiation.cc:769] Sending connection context
0929 19:01:21.313451 (+ 102us) client_negotiation.cc:240] Negotiation successful
0929 19:01:21.313614 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":71,"threads_started":1}
I20260929 19:01:21.314250 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUbnLHE/build/release/bin/kudu as pid 8271
I20260929 19:01:21.314370 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708447806542-392-0/minicluster-data/ts-2/wal/instance
I20260929 19:01:21.314594 392 external_mini_cluster.cc:1428] Setting key d9ea331bd7998e5766a9b0836bc86faa
I20260929 19:01:21.314939 8402 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34221
I20260929 19:01:21.315022 8402 heartbeater.cc:464] Registering TS with master...
I20260929 19:01:21.315222 8402 heartbeater.cc:511] Master 127.0.98.62:34221 requested a full tablet report, sending...
I20260929 19:01:21.315517 7919 ts_manager.cc:194] Registered new tserver with Master: 649983f98f554789af84afcb77b8e38a (127.0.98.3:46835)
I20260929 19:01:21.315994 7919 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:57579
I20260929 19:01:21.320788 7968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:21.315669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57963 (local address 127.0.98.62:34221)
0929 19:01:21.315710 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:21.315712 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:21.315867 (+ 155us) server_negotiation.cc:407] Connection header received
0929 19:01:21.315903 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:21.315904 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:21.315911 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:21.315950 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:01:21.316656 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:21.317168 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:21.317910 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:21.318100 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:21.318154 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:21.318958 (+ 804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:01:21.318958 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:01:21.318959 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:01:21.318963 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:01:21.320025 (+ 1062us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:21.320286 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:21.320287 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:21.320287 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:21.320325 (+ 38us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:21.320454 (+ 129us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:21.320454 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:21.320455 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:21.320535 (+ 80us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:01:21.320578 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0929 19:01:21.320685 (+ 107us) server_negotiation.cc:299] Negotiation successful
0929 19:01:21.320733 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260929 19:01:21.320837 8408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:21.315673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34221 (local address 127.0.98.3:57963)
0929 19:01:21.315792 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:21.315803 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 19:01:21.315859 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:01:21.315961 (+ 102us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:01:21.315962 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:01:21.316170 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:01:21.316231 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:21.316232 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:21.317268 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:21.317269 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:21.317844 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:21.317846 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:21.318163 (+ 317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:21.318163 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:21.318218 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:21.318636 (+ 418us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:01:21.318646 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:01:21.318891 (+ 245us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:01:21.320090 (+ 1199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:01:21.320091 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:01:21.320091 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:01:21.320239 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:01:21.320372 (+ 133us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:01:21.320372 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:01:21.320373 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:01:21.320419 (+ 46us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:01:21.320587 (+ 168us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:01:21.320588 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:01:21.320619 (+ 31us) client_negotiation.cc:769] Sending connection context
0929 19:01:21.320669 (+ 50us) client_negotiation.cc:240] Negotiation successful
0929 19:01:21.320736 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":81,"threads_started":1}
I20260929 19:01:21.326191 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 19:01:21 dist-test-slave-lqkv krb5kdc[7874](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708481, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 19:01:21 dist-test-slave-lqkv krb5kdc[7874](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 19:01:21 dist-test-slave-lqkv krb5kdc[7874](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260929 19:01:21.346131 7968 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:01:21.341212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38768 (local address 127.0.98.62:34221)
0929 19:01:21.341284 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:21.341285 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:21.341294 (+ 9us) server_negotiation.cc:407] Connection header received
0929 19:01:21.341313 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:21.341313 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:21.341321 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:21.341370 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:01:21.341910 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:21.342373 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:21.343051 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:21.343221 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:21.343268 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:21.346031 (+ 2763us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38768: BlockingRecv error: recv got EOF from 127.0.0.1:38768 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 29 19:01:21 dist-test-slave-lqkv krb5kdc[7874](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708481, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260929 19:01:21.352978 7968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:21.346923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38770 (local address 127.0.98.62:34221)
0929 19:01:21.346955 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:21.346957 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:21.347040 (+ 83us) server_negotiation.cc:407] Connection header received
0929 19:01:21.347094 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:21.347095 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:21.347102 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:21.347149 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:01:21.347608 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:21.348031 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:21.348663 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:21.348879 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:21.348933 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:21.350869 (+ 1936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:01:21.350870 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:01:21.350871 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:01:21.350876 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:01:21.352021 (+ 1145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:21.352338 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:21.352338 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:21.352339 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:21.352385 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:21.352537 (+ 152us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:21.352538 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:21.352538 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:21.352628 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:01:21.352698 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 19:01:21.352776 (+ 78us) server_negotiation.cc:299] Negotiation successful
0929 19:01:21.352922 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260929 19:01:21.354452 7919 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38770:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 19:01:21.356010 7919 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 19:01:21.362663 8425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:21.358160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37753 (local address 127.0.0.1:54914)
0929 19:01:21.358622 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:21.358630 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0929 19:01:21.358675 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:01:21.358853 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:01:21.358854 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:01:21.358856 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:01:21.358897 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:21.358898 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:21.360369 (+ 1471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:21.360370 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:21.361368 (+ 998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:21.361370 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:21.362320 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:21.362321 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:21.362401 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:21.362420 (+ 19us) client_negotiation.cc:769] Sending connection context
0929 19:01:21.362496 (+ 76us) client_negotiation.cc:240] Negotiation successful
0929 19:01:21.362581 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":448,"mutex_wait_us":11,"thread_start_us":91,"threads_started":1}
I20260929 19:01:21.362661 8426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:21.358155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46835 (local address 127.0.0.1:42396)
0929 19:01:21.358402 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:21.358438 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0929 19:01:21.358525 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:01:21.358780 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:01:21.358783 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:01:21.358785 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:01:21.358846 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:21.358848 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:21.360302 (+ 1454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:21.360303 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:21.361457 (+ 1154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:21.361460 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:21.362320 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:21.362321 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:21.362401 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:21.362420 (+ 19us) client_negotiation.cc:769] Sending connection context
0929 19:01:21.362496 (+ 76us) client_negotiation.cc:240] Negotiation successful
0929 19:01:21.362571 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"mutex_wait_us":40,"thread_start_us":58,"threads_started":1}
I20260929 19:01:21.363078 8428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:21.358388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54914 (local address 127.0.98.1:37753)
0929 19:01:21.358577 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:21.358582 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:21.358685 (+ 103us) server_negotiation.cc:407] Connection header received
0929 19:01:21.358745 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:21.358747 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:21.358765 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:21.358849 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:01:21.359134 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:21.360271 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:21.361476 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:21.362232 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:21.362311 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:21.362399 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0929 19:01:21.362611 (+ 212us) server_negotiation.cc:299] Negotiation successful
0929 19:01:21.362658 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":88,"threads_started":1}
I20260929 19:01:21.363445 8427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:21.358253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42396 (local address 127.0.98.3:46835)
0929 19:01:21.358620 (+ 367us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:21.358624 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:21.358634 (+ 10us) server_negotiation.cc:407] Connection header received
0929 19:01:21.358664 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:21.358666 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:21.358679 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:21.358775 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:01:21.358929 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:21.360219 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:21.361534 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:21.362232 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:21.362311 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:21.362387 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 19:01:21.363293 (+ 906us) server_negotiation.cc:299] Negotiation successful
0929 19:01:21.363350 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":86,"threads_started":1}
I20260929 19:01:21.363749 8424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:21.358167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37627 (local address 127.0.0.1:60092)
0929 19:01:21.358747 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:21.358756 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 19:01:21.358794 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:01:21.359113 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:01:21.359114 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:01:21.359116 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:01:21.359159 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:21.359161 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:21.359992 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:21.359993 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:21.361470 (+ 1477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:21.361472 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:21.363505 (+ 2033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:21.363506 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:21.363570 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:21.363574 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 19:01:21.363635 (+ 61us) client_negotiation.cc:240] Negotiation successful
0929 19:01:21.363683 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":573,"mutex_wait_us":28,"thread_start_us":69,"threads_started":1}
I20260929 19:01:21.363816 8429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:21.358414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60092 (local address 127.0.98.2:37627)
0929 19:01:21.358907 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:21.358911 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:21.358921 (+ 10us) server_negotiation.cc:407] Connection header received
0929 19:01:21.358957 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:21.358960 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:21.358975 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:21.359051 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:01:21.359237 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:21.359906 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:21.362562 (+ 2656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:21.363307 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:21.363371 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:21.363448 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0929 19:01:21.363669 (+ 221us) server_negotiation.cc:299] Negotiation successful
0929 19:01:21.363712 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":459,"thread_start_us":80,"threads_started":1}
I20260929 19:01:21.363816 8056 tablet_service.cc:1467] Processing CreateTablet for tablet 6ef889d354cd4ee89c00245882428254 (DEFAULT_TABLE table=test-table [id=46a1a9e50bd74510962bd321c7545eec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:01:21.364074 8336 tablet_service.cc:1467] Processing CreateTablet for tablet 6ef889d354cd4ee89c00245882428254 (DEFAULT_TABLE table=test-table [id=46a1a9e50bd74510962bd321c7545eec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:01:21.364181 8056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ef889d354cd4ee89c00245882428254. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:01:21.364308 8336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ef889d354cd4ee89c00245882428254. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:01:21.365505 8196 tablet_service.cc:1467] Processing CreateTablet for tablet 6ef889d354cd4ee89c00245882428254 (DEFAULT_TABLE table=test-table [id=46a1a9e50bd74510962bd321c7545eec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:01:21.365801 8196 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ef889d354cd4ee89c00245882428254. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:01:21.367659 8430 tablet_bootstrap.cc:492] T 6ef889d354cd4ee89c00245882428254 P 936984d765f745738bea4acbbe9c61fb: Bootstrap starting.
I20260929 19:01:21.368233 8430 tablet_bootstrap.cc:654] T 6ef889d354cd4ee89c00245882428254 P 936984d765f745738bea4acbbe9c61fb: Neither blocks nor log segments found. Creating new log.
I20260929 19:01:21.368530 8430 log.cc:824] T 6ef889d354cd4ee89c00245882428254 P 936984d765f745738bea4acbbe9c61fb: Log is configured to *not* fsync() on all Append() calls
I20260929 19:01:21.368875 8431 tablet_bootstrap.cc:492] T 6ef889d354cd4ee89c00245882428254 P 649983f98f554789af84afcb77b8e38a: Bootstrap starting.
I20260929 19:01:21.369347 8430 tablet_bootstrap.cc:492] T 6ef889d354cd4ee89c00245882428254 P 936984d765f745738bea4acbbe9c61fb: No bootstrap required, opened a new log
I20260929 19:01:21.369385 8433 tablet_bootstrap.cc:492] T 6ef889d354cd4ee89c00245882428254 P b27ba18ec1f24f4f9691a5e7de5ff564: Bootstrap starting.
I20260929 19:01:21.369416 8430 ts_tablet_manager.cc:1397] T 6ef889d354cd4ee89c00245882428254 P 936984d765f745738bea4acbbe9c61fb: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 19:01:21.369577 8431 tablet_bootstrap.cc:654] T 6ef889d354cd4ee89c00245882428254 P 649983f98f554789af84afcb77b8e38a: Neither blocks nor log segments found. Creating new log.
I20260929 19:01:21.369845 8433 tablet_bootstrap.cc:654] T 6ef889d354cd4ee89c00245882428254 P b27ba18ec1f24f4f9691a5e7de5ff564: Neither blocks nor log segments found. Creating new log.
I20260929 19:01:21.370057 8433 log.cc:824] T 6ef889d354cd4ee89c00245882428254 P b27ba18ec1f24f4f9691a5e7de5ff564: Log is configured to *not* fsync() on all Append() calls
I20260929 19:01:21.370222 8431 log.cc:824] T 6ef889d354cd4ee89c00245882428254 P 649983f98f554789af84afcb77b8e38a: Log is configured to *not* fsync() on all Append() calls
I20260929 19:01:21.370699 8433 tablet_bootstrap.cc:492] T 6ef889d354cd4ee89c00245882428254 P b27ba18ec1f24f4f9691a5e7de5ff564: No bootstrap required, opened a new log
I20260929 19:01:21.370740 8433 ts_tablet_manager.cc:1397] T 6ef889d354cd4ee89c00245882428254 P b27ba18ec1f24f4f9691a5e7de5ff564: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260929 19:01:21.370899 8431 tablet_bootstrap.cc:492] T 6ef889d354cd4ee89c00245882428254 P 649983f98f554789af84afcb77b8e38a: No bootstrap required, opened a new log
I20260929 19:01:21.370985 8431 ts_tablet_manager.cc:1397] T 6ef889d354cd4ee89c00245882428254 P 649983f98f554789af84afcb77b8e38a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 19:01:21.371676 8430 raft_consensus.cc:348] T 6ef889d354cd4ee89c00245882428254 P 936984d765f745738bea4acbbe9c61fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "649983f98f554789af84afcb77b8e38a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46835 } } peers { permanent_uuid: "936984d765f745738bea4acbbe9c61fb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37753 } } peers { permanent_uuid: "b27ba18ec1f24f4f9691a5e7de5ff564" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37627 } }
I20260929 19:01:21.371835 8430 raft_consensus.cc:374] T 6ef889d354cd4ee89c00245882428254 P 936984d765f745738bea4acbbe9c61fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:01:21.371870 8430 raft_consensus.cc:729] T 6ef889d354cd4ee89c00245882428254 P 936984d765f745738bea4acbbe9c61fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 936984d765f745738bea4acbbe9c61fb, State: Initialized, Role: FOLLOWER
I20260929 19:01:21.371980 8430 consensus_queue.cc:260] T 6ef889d354cd4ee89c00245882428254 P 936984d765f745738bea4acbbe9c61fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "649983f98f554789af84afcb77b8e38a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46835 } } peers { permanent_uuid: "936984d765f745738bea4acbbe9c61fb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37753 } } peers { permanent_uuid: "b27ba18ec1f24f4f9691a5e7de5ff564" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37627 } }
I20260929 19:01:21.372012 8433 raft_consensus.cc:348] T 6ef889d354cd4ee89c00245882428254 P b27ba18ec1f24f4f9691a5e7de5ff564 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "649983f98f554789af84afcb77b8e38a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46835 } } peers { permanent_uuid: "936984d765f745738bea4acbbe9c61fb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37753 } } peers { permanent_uuid: "b27ba18ec1f24f4f9691a5e7de5ff564" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37627 } }
I20260929 19:01:21.372134 8433 raft_consensus.cc:374] T 6ef889d354cd4ee89c00245882428254 P b27ba18ec1f24f4f9691a5e7de5ff564 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:01:21.372181 8433 raft_consensus.cc:729] T 6ef889d354cd4ee89c00245882428254 P b27ba18ec1f24f4f9691a5e7de5ff564 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b27ba18ec1f24f4f9691a5e7de5ff564, State: Initialized, Role: FOLLOWER
I20260929 19:01:21.372236 8430 ts_tablet_manager.cc:1428] T 6ef889d354cd4ee89c00245882428254 P 936984d765f745738bea4acbbe9c61fb: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 19:01:21.372300 8433 consensus_queue.cc:260] T 6ef889d354cd4ee89c00245882428254 P b27ba18ec1f24f4f9691a5e7de5ff564 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "649983f98f554789af84afcb77b8e38a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46835 } } peers { permanent_uuid: "936984d765f745738bea4acbbe9c61fb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37753 } } peers { permanent_uuid: "b27ba18ec1f24f4f9691a5e7de5ff564" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37627 } }
I20260929 19:01:21.372550 8433 ts_tablet_manager.cc:1428] T 6ef889d354cd4ee89c00245882428254 P b27ba18ec1f24f4f9691a5e7de5ff564: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 19:01:21.372619 8262 heartbeater.cc:503] Master 127.0.98.62:34221 was elected leader, sending a full tablet report...
I20260929 19:01:21.373011 8431 raft_consensus.cc:348] T 6ef889d354cd4ee89c00245882428254 P 649983f98f554789af84afcb77b8e38a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "649983f98f554789af84afcb77b8e38a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46835 } } peers { permanent_uuid: "936984d765f745738bea4acbbe9c61fb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37753 } } peers { permanent_uuid: "b27ba18ec1f24f4f9691a5e7de5ff564" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37627 } }
I20260929 19:01:21.373153 8431 raft_consensus.cc:374] T 6ef889d354cd4ee89c00245882428254 P 649983f98f554789af84afcb77b8e38a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:01:21.373185 8431 raft_consensus.cc:729] T 6ef889d354cd4ee89c00245882428254 P 649983f98f554789af84afcb77b8e38a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 649983f98f554789af84afcb77b8e38a, State: Initialized, Role: FOLLOWER
I20260929 19:01:21.373292 8431 consensus_queue.cc:260] T 6ef889d354cd4ee89c00245882428254 P 649983f98f554789af84afcb77b8e38a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "649983f98f554789af84afcb77b8e38a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46835 } } peers { permanent_uuid: "936984d765f745738bea4acbbe9c61fb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37753 } } peers { permanent_uuid: "b27ba18ec1f24f4f9691a5e7de5ff564" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37627 } }
I20260929 19:01:21.373534 8431 ts_tablet_manager.cc:1428] T 6ef889d354cd4ee89c00245882428254 P 649983f98f554789af84afcb77b8e38a: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260929 19:01:21.373600 8402 heartbeater.cc:503] Master 127.0.98.62:34221 was elected leader, sending a full tablet report...
I20260929 19:01:21.373718 8122 heartbeater.cc:503] Master 127.0.98.62:34221 was elected leader, sending a full tablet report...
W20260929 19:01:21.406442 8263 tablet.cc:2369] T 6ef889d354cd4ee89c00245882428254 P b27ba18ec1f24f4f9691a5e7de5ff564: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 19:01:21.450804 8438 raft_consensus.cc:482] T 6ef889d354cd4ee89c00245882428254 P 649983f98f554789af84afcb77b8e38a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 19:01:21.450919 8438 raft_consensus.cc:504] T 6ef889d354cd4ee89c00245882428254 P 649983f98f554789af84afcb77b8e38a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "649983f98f554789af84afcb77b8e38a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46835 } } peers { permanent_uuid: "936984d765f745738bea4acbbe9c61fb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37753 } } peers { permanent_uuid: "b27ba18ec1f24f4f9691a5e7de5ff564" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37627 } }
I20260929 19:01:21.451375 8438 leader_election.cc:288] T 6ef889d354cd4ee89c00245882428254 P 649983f98f554789af84afcb77b8e38a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 936984d765f745738bea4acbbe9c61fb (127.0.98.1:37753), b27ba18ec1f24f4f9691a5e7de5ff564 (127.0.98.2:37627)
I20260929 19:01:21.455299 8428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:21.451641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56731 (local address 127.0.98.1:37753)
0929 19:01:21.451713 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:21.451715 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:21.451753 (+ 38us) server_negotiation.cc:407] Connection header received
0929 19:01:21.451781 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:21.451782 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:21.451792 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:21.451843 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:01:21.452022 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:21.452796 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:21.454369 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:21.454922 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:21.454999 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:21.455008 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0929 19:01:21.455192 (+ 184us) server_negotiation.cc:299] Negotiation successful
0929 19:01:21.455231 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 19:01:21.455299 8429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:21.451707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34799 (local address 127.0.98.2:37627)
0929 19:01:21.451750 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:21.451752 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:21.451905 (+ 153us) server_negotiation.cc:407] Connection header received
0929 19:01:21.451939 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:21.451940 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:21.451948 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:21.452015 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:01:21.452146 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:21.452860 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:21.454369 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:21.454923 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:21.454999 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:21.455008 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0929 19:01:21.455192 (+ 184us) server_negotiation.cc:299] Negotiation successful
0929 19:01:21.455234 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260929 19:01:21.455303 8295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:21.451570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37753 (local address 127.0.98.3:56731)
0929 19:01:21.451648 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:21.451661 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 19:01:21.451710 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:01:21.451870 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:01:21.451871 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:01:21.451873 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:01:21.451946 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:21.451947 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:21.452909 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:21.452910 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:21.454271 (+ 1361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:21.454275 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:21.455010 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:21.455011 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:21.455095 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:21.455106 (+ 11us) client_negotiation.cc:769] Sending connection context
0929 19:01:21.455167 (+ 61us) client_negotiation.cc:240] Negotiation successful
0929 19:01:21.455210 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55,"mutex_wait_us":98}
I20260929 19:01:21.455308 8439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:21.451579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37627 (local address 127.0.98.3:34799)
0929 19:01:21.451829 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:21.451838 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 19:01:21.451886 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:01:21.452016 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:01:21.452017 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:01:21.452019 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:01:21.452084 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:21.452085 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:21.452929 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:21.452930 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:21.454271 (+ 1341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:21.454275 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:21.455011 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:21.455011 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:21.455109 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:21.455112 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 19:01:21.455167 (+ 55us) client_negotiation.cc:240] Negotiation successful
0929 19:01:21.455210 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"mutex_wait_us":139,"thread_start_us":111,"threads_started":1}
I20260929 19:01:21.455739 8076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ef889d354cd4ee89c00245882428254" candidate_uuid: "649983f98f554789af84afcb77b8e38a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "936984d765f745738bea4acbbe9c61fb" is_pre_election: true
I20260929 19:01:21.455916 8076 raft_consensus.cc:2454] T 6ef889d354cd4ee89c00245882428254 P 936984d765f745738bea4acbbe9c61fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 649983f98f554789af84afcb77b8e38a in term 0.
I20260929 19:01:21.455904 8205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ef889d354cd4ee89c00245882428254" candidate_uuid: "649983f98f554789af84afcb77b8e38a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b27ba18ec1f24f4f9691a5e7de5ff564" is_pre_election: true
I20260929 19:01:21.456076 8205 raft_consensus.cc:2454] T 6ef889d354cd4ee89c00245882428254 P b27ba18ec1f24f4f9691a5e7de5ff564 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 649983f98f554789af84afcb77b8e38a in term 0.
I20260929 19:01:21.456183 8289 leader_election.cc:302] T 6ef889d354cd4ee89c00245882428254 P 649983f98f554789af84afcb77b8e38a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 649983f98f554789af84afcb77b8e38a, 936984d765f745738bea4acbbe9c61fb; no voters:
I20260929 19:01:21.456312 8438 raft_consensus.cc:2781] T 6ef889d354cd4ee89c00245882428254 P 649983f98f554789af84afcb77b8e38a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 19:01:21.456386 8438 raft_consensus.cc:482] T 6ef889d354cd4ee89c00245882428254 P 649983f98f554789af84afcb77b8e38a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 19:01:21.456404 8438 raft_consensus.cc:3037] T 6ef889d354cd4ee89c00245882428254 P 649983f98f554789af84afcb77b8e38a [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:01:21.457361 8438 raft_consensus.cc:504] T 6ef889d354cd4ee89c00245882428254 P 649983f98f554789af84afcb77b8e38a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "649983f98f554789af84afcb77b8e38a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46835 } } peers { permanent_uuid: "936984d765f745738bea4acbbe9c61fb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37753 } } peers { permanent_uuid: "b27ba18ec1f24f4f9691a5e7de5ff564" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37627 } }
I20260929 19:01:21.457504 8438 leader_election.cc:288] T 6ef889d354cd4ee89c00245882428254 P 649983f98f554789af84afcb77b8e38a [CANDIDATE]: Term 1 election: Requested vote from peers 936984d765f745738bea4acbbe9c61fb (127.0.98.1:37753), b27ba18ec1f24f4f9691a5e7de5ff564 (127.0.98.2:37627)
I20260929 19:01:21.457649 8076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ef889d354cd4ee89c00245882428254" candidate_uuid: "649983f98f554789af84afcb77b8e38a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "936984d765f745738bea4acbbe9c61fb"
I20260929 19:01:21.457681 8205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ef889d354cd4ee89c00245882428254" candidate_uuid: "649983f98f554789af84afcb77b8e38a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b27ba18ec1f24f4f9691a5e7de5ff564"
I20260929 19:01:21.457746 8076 raft_consensus.cc:3037] T 6ef889d354cd4ee89c00245882428254 P 936984d765f745738bea4acbbe9c61fb [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:01:21.457743 8205 raft_consensus.cc:3037] T 6ef889d354cd4ee89c00245882428254 P b27ba18ec1f24f4f9691a5e7de5ff564 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:01:21.458418 8205 raft_consensus.cc:2454] T 6ef889d354cd4ee89c00245882428254 P b27ba18ec1f24f4f9691a5e7de5ff564 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 649983f98f554789af84afcb77b8e38a in term 1.
I20260929 19:01:21.458501 8076 raft_consensus.cc:2454] T 6ef889d354cd4ee89c00245882428254 P 936984d765f745738bea4acbbe9c61fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 649983f98f554789af84afcb77b8e38a in term 1.
I20260929 19:01:21.458595 8290 leader_election.cc:302] T 6ef889d354cd4ee89c00245882428254 P 649983f98f554789af84afcb77b8e38a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 649983f98f554789af84afcb77b8e38a, b27ba18ec1f24f4f9691a5e7de5ff564; no voters:
I20260929 19:01:21.458693 8438 raft_consensus.cc:2781] T 6ef889d354cd4ee89c00245882428254 P 649983f98f554789af84afcb77b8e38a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:01:21.458866 8438 raft_consensus.cc:686] T 6ef889d354cd4ee89c00245882428254 P 649983f98f554789af84afcb77b8e38a [term 1 LEADER]: Becoming Leader. State: Replica: 649983f98f554789af84afcb77b8e38a, State: Running, Role: LEADER
I20260929 19:01:21.458966 8438 consensus_queue.cc:237] T 6ef889d354cd4ee89c00245882428254 P 649983f98f554789af84afcb77b8e38a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "649983f98f554789af84afcb77b8e38a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46835 } } peers { permanent_uuid: "936984d765f745738bea4acbbe9c61fb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37753 } } peers { permanent_uuid: "b27ba18ec1f24f4f9691a5e7de5ff564" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37627 } }
I20260929 19:01:21.459776 7917 catalog_manager.cc:5495] T 6ef889d354cd4ee89c00245882428254 P 649983f98f554789af84afcb77b8e38a reported cstate change: term changed from 0 to 1, leader changed from <none> to 649983f98f554789af84afcb77b8e38a (127.0.98.3). New cstate: current_term: 1 leader_uuid: "649983f98f554789af84afcb77b8e38a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "649983f98f554789af84afcb77b8e38a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46835 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "936984d765f745738bea4acbbe9c61fb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37753 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b27ba18ec1f24f4f9691a5e7de5ff564" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37627 } health_report { overall_health: UNKNOWN } } }
I20260929 19:01:21.476725 8427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:21.474628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42406 (local address 127.0.98.3:46835)
0929 19:01:21.474689 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:21.474691 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:21.474701 (+ 10us) server_negotiation.cc:407] Connection header received
0929 19:01:21.474731 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:21.474732 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:21.474755 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:21.474816 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 19:01:21.474911 (+ 95us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:21.475362 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:21.476063 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:21.476246 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:21.476295 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:21.476420 (+ 125us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 19:01:21.476503 (+ 83us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 19:01:21.476559 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0929 19:01:21.476603 (+ 44us) server_negotiation.cc:299] Negotiation successful
0929 19:01:21.476622 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 19:01:21.478398 8076 raft_consensus.cc:1261] T 6ef889d354cd4ee89c00245882428254 P 936984d765f745738bea4acbbe9c61fb [term 1 FOLLOWER]: Refusing update from remote peer 649983f98f554789af84afcb77b8e38a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 19:01:21.478468 8205 raft_consensus.cc:1261] T 6ef889d354cd4ee89c00245882428254 P b27ba18ec1f24f4f9691a5e7de5ff564 [term 1 FOLLOWER]: Refusing update from remote peer 649983f98f554789af84afcb77b8e38a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 19:01:21.478802 8440 consensus_queue.cc:1035] T 6ef889d354cd4ee89c00245882428254 P 649983f98f554789af84afcb77b8e38a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b27ba18ec1f24f4f9691a5e7de5ff564" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37627 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 19:01:21.478906 8438 consensus_queue.cc:1035] T 6ef889d354cd4ee89c00245882428254 P 649983f98f554789af84afcb77b8e38a [LEADER]: Connected to new peer: Peer: permanent_uuid: "936984d765f745738bea4acbbe9c61fb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 19:01:21.482084 8445 mvcc.cc:205] Tried to move back new op lower bound from 7334741940132577280 to 7334741940056297472. Current Snapshot: MvccSnapshot[applied={T|T < 7334741940132577280}]
I20260929 19:01:21.482344 8444 mvcc.cc:205] Tried to move back new op lower bound from 7334741940132577280 to 7334741940056297472. Current Snapshot: MvccSnapshot[applied={T|T < 7334741940132577280 or (T in {7334741940132577280})}]
I20260929 19:01:21.483428 7917 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38770:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 19:01:21.483529 7917 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38770:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 19:01:21.484261 7917 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1db5b3fcae274bbf94cb5d9e6eff70fa: Sending DeleteTablet for 3 replicas of tablet 6ef889d354cd4ee89c00245882428254
I20260929 19:01:21.484568 8336 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ef889d354cd4ee89c00245882428254 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:01:21 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42396
I20260929 19:01:21.484740 8196 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ef889d354cd4ee89c00245882428254 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:01:21 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60092
I20260929 19:01:21.484973 8454 tablet_replica.cc:330] T 6ef889d354cd4ee89c00245882428254 P b27ba18ec1f24f4f9691a5e7de5ff564: stopping tablet replica
I20260929 19:01:21.485073 8454 raft_consensus.cc:2229] T 6ef889d354cd4ee89c00245882428254 P b27ba18ec1f24f4f9691a5e7de5ff564 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 19:01:21.485210 8454 raft_consensus.cc:2258] T 6ef889d354cd4ee89c00245882428254 P b27ba18ec1f24f4f9691a5e7de5ff564 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 19:01:21.485268 8056 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ef889d354cd4ee89c00245882428254 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:01:21 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54914
I20260929 19:01:21.485248 7917 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38770:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 19:01:21.485424 7917 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 19:01:21.485497 8455 tablet_replica.cc:330] T 6ef889d354cd4ee89c00245882428254 P 936984d765f745738bea4acbbe9c61fb: stopping tablet replica
I20260929 19:01:21.485709 8455 raft_consensus.cc:2229] T 6ef889d354cd4ee89c00245882428254 P 936984d765f745738bea4acbbe9c61fb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 19:01:21.485822 8455 raft_consensus.cc:2258] T 6ef889d354cd4ee89c00245882428254 P 936984d765f745738bea4acbbe9c61fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 19:01:21.485849 8454 ts_tablet_manager.cc:1905] T 6ef889d354cd4ee89c00245882428254 P b27ba18ec1f24f4f9691a5e7de5ff564: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 19:01:21.486256 8455 ts_tablet_manager.cc:1905] T 6ef889d354cd4ee89c00245882428254 P 936984d765f745738bea4acbbe9c61fb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 19:01:21.487036 8336 tablet_service.cc:1467] Processing CreateTablet for tablet f0349ee8b2d240caa92e259ca18b9d0f (DEFAULT_TABLE table=test-table [id=9e35b529267e4e21aab6dfefec8b997c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:01:21.487151 8336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0349ee8b2d240caa92e259ca18b9d0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:01:21.487471 8454 ts_tablet_manager.cc:1918] T 6ef889d354cd4ee89c00245882428254 P b27ba18ec1f24f4f9691a5e7de5ff564: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 19:01:21.487546 8454 log.cc:1196] T 6ef889d354cd4ee89c00245882428254 P b27ba18ec1f24f4f9691a5e7de5ff564: Deleting WAL directory at /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708447806542-392-0/minicluster-data/ts-1/wal/wals/6ef889d354cd4ee89c00245882428254
I20260929 19:01:21.487797 8454 ts_tablet_manager.cc:1939] T 6ef889d354cd4ee89c00245882428254 P b27ba18ec1f24f4f9691a5e7de5ff564: Deleting consensus metadata
I20260929 19:01:21.488260 7906 catalog_manager.cc:4841] TS b27ba18ec1f24f4f9691a5e7de5ff564 (127.0.98.2:37627): tablet 6ef889d354cd4ee89c00245882428254 (table test-table [id=46a1a9e50bd74510962bd321c7545eec]) successfully deleted
I20260929 19:01:21.488641 8431 tablet_bootstrap.cc:492] T f0349ee8b2d240caa92e259ca18b9d0f P 649983f98f554789af84afcb77b8e38a: Bootstrap starting.
I20260929 19:01:21.488907 8453 tablet_replica.cc:330] T 6ef889d354cd4ee89c00245882428254 P 649983f98f554789af84afcb77b8e38a: stopping tablet replica
I20260929 19:01:21.489044 8453 raft_consensus.cc:2229] T 6ef889d354cd4ee89c00245882428254 P 649983f98f554789af84afcb77b8e38a [term 1 LEADER]: Raft consensus shutting down.
I20260929 19:01:21.489059 8455 ts_tablet_manager.cc:1918] T 6ef889d354cd4ee89c00245882428254 P 936984d765f745738bea4acbbe9c61fb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 19:01:21.489137 8455 log.cc:1196] T 6ef889d354cd4ee89c00245882428254 P 936984d765f745738bea4acbbe9c61fb: Deleting WAL directory at /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708447806542-392-0/minicluster-data/ts-0/wal/wals/6ef889d354cd4ee89c00245882428254
I20260929 19:01:21.489260 8453 raft_consensus.cc:2258] T 6ef889d354cd4ee89c00245882428254 P 649983f98f554789af84afcb77b8e38a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 19:01:21.489420 8455 ts_tablet_manager.cc:1939] T 6ef889d354cd4ee89c00245882428254 P 936984d765f745738bea4acbbe9c61fb: Deleting consensus metadata
I20260929 19:01:21.489506 8196 tablet_service.cc:1467] Processing CreateTablet for tablet f0349ee8b2d240caa92e259ca18b9d0f (DEFAULT_TABLE table=test-table [id=9e35b529267e4e21aab6dfefec8b997c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:01:21.489604 8196 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0349ee8b2d240caa92e259ca18b9d0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:01:21.489800 8453 ts_tablet_manager.cc:1905] T 6ef889d354cd4ee89c00245882428254 P 649983f98f554789af84afcb77b8e38a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 19:01:21.490500 8433 tablet_bootstrap.cc:492] T f0349ee8b2d240caa92e259ca18b9d0f P b27ba18ec1f24f4f9691a5e7de5ff564: Bootstrap starting.
I20260929 19:01:21.489401 8056 tablet_service.cc:1467] Processing CreateTablet for tablet f0349ee8b2d240caa92e259ca18b9d0f (DEFAULT_TABLE table=test-table [id=9e35b529267e4e21aab6dfefec8b997c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:01:21.490762 8056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0349ee8b2d240caa92e259ca18b9d0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:01:21.490891 8433 tablet_bootstrap.cc:654] T f0349ee8b2d240caa92e259ca18b9d0f P b27ba18ec1f24f4f9691a5e7de5ff564: Neither blocks nor log segments found. Creating new log.
I20260929 19:01:21.491420 8453 ts_tablet_manager.cc:1918] T 6ef889d354cd4ee89c00245882428254 P 649983f98f554789af84afcb77b8e38a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 19:01:21.491536 8453 log.cc:1196] T 6ef889d354cd4ee89c00245882428254 P 649983f98f554789af84afcb77b8e38a: Deleting WAL directory at /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708447806542-392-0/minicluster-data/ts-2/wal/wals/6ef889d354cd4ee89c00245882428254
I20260929 19:01:21.491799 8430 tablet_bootstrap.cc:492] T f0349ee8b2d240caa92e259ca18b9d0f P 936984d765f745738bea4acbbe9c61fb: Bootstrap starting.
I20260929 19:01:21.491868 7905 catalog_manager.cc:4841] TS 936984d765f745738bea4acbbe9c61fb (127.0.98.1:37753): tablet 6ef889d354cd4ee89c00245882428254 (table test-table [id=46a1a9e50bd74510962bd321c7545eec]) successfully deleted
I20260929 19:01:21.491890 8453 ts_tablet_manager.cc:1939] T 6ef889d354cd4ee89c00245882428254 P 649983f98f554789af84afcb77b8e38a: Deleting consensus metadata
I20260929 19:01:21.489070 8431 tablet_bootstrap.cc:654] T f0349ee8b2d240caa92e259ca18b9d0f P 649983f98f554789af84afcb77b8e38a: Neither blocks nor log segments found. Creating new log.
I20260929 19:01:21.492214 8430 tablet_bootstrap.cc:654] T f0349ee8b2d240caa92e259ca18b9d0f P 936984d765f745738bea4acbbe9c61fb: Neither blocks nor log segments found. Creating new log.
I20260929 19:01:21.492419 7903 catalog_manager.cc:4841] TS 649983f98f554789af84afcb77b8e38a (127.0.98.3:46835): tablet 6ef889d354cd4ee89c00245882428254 (table test-table [id=46a1a9e50bd74510962bd321c7545eec]) successfully deleted
I20260929 19:01:21.492772 8431 tablet_bootstrap.cc:492] T f0349ee8b2d240caa92e259ca18b9d0f P 649983f98f554789af84afcb77b8e38a: No bootstrap required, opened a new log
I20260929 19:01:21.492846 8431 ts_tablet_manager.cc:1397] T f0349ee8b2d240caa92e259ca18b9d0f P 649983f98f554789af84afcb77b8e38a: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20260929 19:01:21.492864 8433 tablet_bootstrap.cc:492] T f0349ee8b2d240caa92e259ca18b9d0f P b27ba18ec1f24f4f9691a5e7de5ff564: No bootstrap required, opened a new log
I20260929 19:01:21.492901 8433 ts_tablet_manager.cc:1397] T f0349ee8b2d240caa92e259ca18b9d0f P b27ba18ec1f24f4f9691a5e7de5ff564: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 19:01:21.492906 8430 tablet_bootstrap.cc:492] T f0349ee8b2d240caa92e259ca18b9d0f P 936984d765f745738bea4acbbe9c61fb: No bootstrap required, opened a new log
I20260929 19:01:21.492947 8430 ts_tablet_manager.cc:1397] T f0349ee8b2d240caa92e259ca18b9d0f P 936984d765f745738bea4acbbe9c61fb: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260929 19:01:21.493008 8431 raft_consensus.cc:348] T f0349ee8b2d240caa92e259ca18b9d0f P 649983f98f554789af84afcb77b8e38a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "936984d765f745738bea4acbbe9c61fb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37753 } } peers { permanent_uuid: "b27ba18ec1f24f4f9691a5e7de5ff564" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37627 } } peers { permanent_uuid: "649983f98f554789af84afcb77b8e38a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46835 } }
I20260929 19:01:21.493070 8431 raft_consensus.cc:374] T f0349ee8b2d240caa92e259ca18b9d0f P 649983f98f554789af84afcb77b8e38a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:01:21.493039 8433 raft_consensus.cc:348] T f0349ee8b2d240caa92e259ca18b9d0f P b27ba18ec1f24f4f9691a5e7de5ff564 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "936984d765f745738bea4acbbe9c61fb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37753 } } peers { permanent_uuid: "b27ba18ec1f24f4f9691a5e7de5ff564" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37627 } } peers { permanent_uuid: "649983f98f554789af84afcb77b8e38a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46835 } }
I20260929 19:01:21.493090 8431 raft_consensus.cc:729] T f0349ee8b2d240caa92e259ca18b9d0f P 649983f98f554789af84afcb77b8e38a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 649983f98f554789af84afcb77b8e38a, State: Initialized, Role: FOLLOWER
I20260929 19:01:21.493096 8433 raft_consensus.cc:374] T f0349ee8b2d240caa92e259ca18b9d0f P b27ba18ec1f24f4f9691a5e7de5ff564 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:01:21.493119 8433 raft_consensus.cc:729] T f0349ee8b2d240caa92e259ca18b9d0f P b27ba18ec1f24f4f9691a5e7de5ff564 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b27ba18ec1f24f4f9691a5e7de5ff564, State: Initialized, Role: FOLLOWER
I20260929 19:01:21.493078 8430 raft_consensus.cc:348] T f0349ee8b2d240caa92e259ca18b9d0f P 936984d765f745738bea4acbbe9c61fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "936984d765f745738bea4acbbe9c61fb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37753 } } peers { permanent_uuid: "b27ba18ec1f24f4f9691a5e7de5ff564" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37627 } } peers { permanent_uuid: "649983f98f554789af84afcb77b8e38a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46835 } }
I20260929 19:01:21.493149 8430 raft_consensus.cc:374] T f0349ee8b2d240caa92e259ca18b9d0f P 936984d765f745738bea4acbbe9c61fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:01:21.493171 8430 raft_consensus.cc:729] T f0349ee8b2d240caa92e259ca18b9d0f P 936984d765f745738bea4acbbe9c61fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 936984d765f745738bea4acbbe9c61fb, State: Initialized, Role: FOLLOWER
I20260929 19:01:21.493151 8431 consensus_queue.cc:260] T f0349ee8b2d240caa92e259ca18b9d0f P 649983f98f554789af84afcb77b8e38a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "936984d765f745738bea4acbbe9c61fb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37753 } } peers { permanent_uuid: "b27ba18ec1f24f4f9691a5e7de5ff564" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37627 } } peers { permanent_uuid: "649983f98f554789af84afcb77b8e38a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46835 } }
I20260929 19:01:21.493155 8433 consensus_queue.cc:260] T f0349ee8b2d240caa92e259ca18b9d0f P b27ba18ec1f24f4f9691a5e7de5ff564 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "936984d765f745738bea4acbbe9c61fb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37753 } } peers { permanent_uuid: "b27ba18ec1f24f4f9691a5e7de5ff564" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37627 } } peers { permanent_uuid: "649983f98f554789af84afcb77b8e38a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46835 } }
I20260929 19:01:21.493230 8431 ts_tablet_manager.cc:1428] T f0349ee8b2d240caa92e259ca18b9d0f P 649983f98f554789af84afcb77b8e38a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:21.493233 8433 ts_tablet_manager.cc:1428] T f0349ee8b2d240caa92e259ca18b9d0f P b27ba18ec1f24f4f9691a5e7de5ff564: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:21.493218 8430 consensus_queue.cc:260] T f0349ee8b2d240caa92e259ca18b9d0f P 936984d765f745738bea4acbbe9c61fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "936984d765f745738bea4acbbe9c61fb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37753 } } peers { permanent_uuid: "b27ba18ec1f24f4f9691a5e7de5ff564" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37627 } } peers { permanent_uuid: "649983f98f554789af84afcb77b8e38a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46835 } }
I20260929 19:01:21.493295 8430 ts_tablet_manager.cc:1428] T f0349ee8b2d240caa92e259ca18b9d0f P 936984d765f745738bea4acbbe9c61fb: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
W20260929 19:01:21.499739 8123 tablet.cc:2369] T f0349ee8b2d240caa92e259ca18b9d0f P 936984d765f745738bea4acbbe9c61fb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 19:01:21.523619 8437 raft_consensus.cc:482] T f0349ee8b2d240caa92e259ca18b9d0f P b27ba18ec1f24f4f9691a5e7de5ff564 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 19:01:21.523715 8437 raft_consensus.cc:504] T f0349ee8b2d240caa92e259ca18b9d0f P b27ba18ec1f24f4f9691a5e7de5ff564 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "936984d765f745738bea4acbbe9c61fb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37753 } } peers { permanent_uuid: "b27ba18ec1f24f4f9691a5e7de5ff564" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37627 } } peers { permanent_uuid: "649983f98f554789af84afcb77b8e38a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46835 } }
I20260929 19:01:21.524101 8437 leader_election.cc:288] T f0349ee8b2d240caa92e259ca18b9d0f P b27ba18ec1f24f4f9691a5e7de5ff564 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 936984d765f745738bea4acbbe9c61fb (127.0.98.1:37753), 649983f98f554789af84afcb77b8e38a (127.0.98.3:46835)
I20260929 19:01:21.527617 8427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:21.524454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48549 (local address 127.0.98.3:46835)
0929 19:01:21.524499 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:21.524501 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:21.524509 (+ 8us) server_negotiation.cc:407] Connection header received
0929 19:01:21.524539 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:21.524540 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:21.524549 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:21.524598 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:01:21.524831 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:21.525613 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:21.526838 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:21.527210 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:21.527307 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:21.527318 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0929 19:01:21.527502 (+ 184us) server_negotiation.cc:299] Negotiation successful
0929 19:01:21.527545 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260929 19:01:21.527624 8155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:21.524277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46835 (local address 127.0.98.2:48549)
0929 19:01:21.524346 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:21.524360 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 19:01:21.524407 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:01:21.524662 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:01:21.524663 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:01:21.524664 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:01:21.524746 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:21.524748 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:21.525707 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:21.525707 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:21.526764 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:21.526766 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:21.527308 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:21.527309 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:21.527401 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:21.527404 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 19:01:21.527482 (+ 78us) client_negotiation.cc:240] Negotiation successful
0929 19:01:21.527523 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260929 19:01:21.527745 8428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:21.524324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59247 (local address 127.0.98.1:37753)
0929 19:01:21.524371 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:21.524373 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:21.524620 (+ 247us) server_negotiation.cc:407] Connection header received
0929 19:01:21.524654 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:21.524655 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:21.524664 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:21.524726 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:01:21.524894 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:21.525373 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:21.526807 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:21.527369 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:21.527443 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:21.527452 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0929 19:01:21.527616 (+ 164us) server_negotiation.cc:299] Negotiation successful
0929 19:01:21.527665 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260929 19:01:21.527912 8356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0349ee8b2d240caa92e259ca18b9d0f" candidate_uuid: "b27ba18ec1f24f4f9691a5e7de5ff564" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "649983f98f554789af84afcb77b8e38a" is_pre_election: true
I20260929 19:01:21.527983 8459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:21.524279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37753 (local address 127.0.98.2:59247)
0929 19:01:21.524510 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:21.524520 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 19:01:21.524575 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:01:21.524741 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:01:21.524742 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:01:21.524743 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:01:21.524835 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:21.524836 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:21.525470 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:21.525471 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:21.526717 (+ 1246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:21.526720 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:21.527470 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:21.527471 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:21.527530 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:21.527533 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 19:01:21.527844 (+ 311us) client_negotiation.cc:240] Negotiation successful
0929 19:01:21.527890 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"mutex_wait_us":105,"thread_start_us":67,"threads_started":1}
I20260929 19:01:21.528033 8356 raft_consensus.cc:2454] T f0349ee8b2d240caa92e259ca18b9d0f P 649983f98f554789af84afcb77b8e38a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b27ba18ec1f24f4f9691a5e7de5ff564 in term 0.
I20260929 19:01:21.528170 8076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0349ee8b2d240caa92e259ca18b9d0f" candidate_uuid: "b27ba18ec1f24f4f9691a5e7de5ff564" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "936984d765f745738bea4acbbe9c61fb" is_pre_election: true
I20260929 19:01:21.528208 8147 leader_election.cc:302] T f0349ee8b2d240caa92e259ca18b9d0f P b27ba18ec1f24f4f9691a5e7de5ff564 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 649983f98f554789af84afcb77b8e38a, b27ba18ec1f24f4f9691a5e7de5ff564; no voters:
I20260929 19:01:21.528256 8076 raft_consensus.cc:2454] T f0349ee8b2d240caa92e259ca18b9d0f P 936984d765f745738bea4acbbe9c61fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b27ba18ec1f24f4f9691a5e7de5ff564 in term 0.
I20260929 19:01:21.528322 8437 raft_consensus.cc:2781] T f0349ee8b2d240caa92e259ca18b9d0f P b27ba18ec1f24f4f9691a5e7de5ff564 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 19:01:21.528357 8437 raft_consensus.cc:482] T f0349ee8b2d240caa92e259ca18b9d0f P b27ba18ec1f24f4f9691a5e7de5ff564 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 19:01:21.528410 8437 raft_consensus.cc:3037] T f0349ee8b2d240caa92e259ca18b9d0f P b27ba18ec1f24f4f9691a5e7de5ff564 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:01:21.529299 8437 raft_consensus.cc:504] T f0349ee8b2d240caa92e259ca18b9d0f P b27ba18ec1f24f4f9691a5e7de5ff564 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "936984d765f745738bea4acbbe9c61fb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37753 } } peers { permanent_uuid: "b27ba18ec1f24f4f9691a5e7de5ff564" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37627 } } peers { permanent_uuid: "649983f98f554789af84afcb77b8e38a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46835 } }
I20260929 19:01:21.529433 8437 leader_election.cc:288] T f0349ee8b2d240caa92e259ca18b9d0f P b27ba18ec1f24f4f9691a5e7de5ff564 [CANDIDATE]: Term 1 election: Requested vote from peers 936984d765f745738bea4acbbe9c61fb (127.0.98.1:37753), 649983f98f554789af84afcb77b8e38a (127.0.98.3:46835)
I20260929 19:01:21.529623 8076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0349ee8b2d240caa92e259ca18b9d0f" candidate_uuid: "b27ba18ec1f24f4f9691a5e7de5ff564" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "936984d765f745738bea4acbbe9c61fb"
I20260929 19:01:21.529698 8076 raft_consensus.cc:3037] T f0349ee8b2d240caa92e259ca18b9d0f P 936984d765f745738bea4acbbe9c61fb [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:01:21.529708 8356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0349ee8b2d240caa92e259ca18b9d0f" candidate_uuid: "b27ba18ec1f24f4f9691a5e7de5ff564" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "649983f98f554789af84afcb77b8e38a"
I20260929 19:01:21.529781 8356 raft_consensus.cc:3037] T f0349ee8b2d240caa92e259ca18b9d0f P 649983f98f554789af84afcb77b8e38a [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:01:21.530306 8076 raft_consensus.cc:2454] T f0349ee8b2d240caa92e259ca18b9d0f P 936984d765f745738bea4acbbe9c61fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b27ba18ec1f24f4f9691a5e7de5ff564 in term 1.
I20260929 19:01:21.530475 8149 leader_election.cc:302] T f0349ee8b2d240caa92e259ca18b9d0f P b27ba18ec1f24f4f9691a5e7de5ff564 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 936984d765f745738bea4acbbe9c61fb, b27ba18ec1f24f4f9691a5e7de5ff564; no voters:
I20260929 19:01:21.530548 8356 raft_consensus.cc:2454] T f0349ee8b2d240caa92e259ca18b9d0f P 649983f98f554789af84afcb77b8e38a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b27ba18ec1f24f4f9691a5e7de5ff564 in term 1.
I20260929 19:01:21.530725 8437 raft_consensus.cc:2781] T f0349ee8b2d240caa92e259ca18b9d0f P b27ba18ec1f24f4f9691a5e7de5ff564 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:01:21.530869 8437 raft_consensus.cc:686] T f0349ee8b2d240caa92e259ca18b9d0f P b27ba18ec1f24f4f9691a5e7de5ff564 [term 1 LEADER]: Becoming Leader. State: Replica: b27ba18ec1f24f4f9691a5e7de5ff564, State: Running, Role: LEADER
I20260929 19:01:21.530936 8437 consensus_queue.cc:237] T f0349ee8b2d240caa92e259ca18b9d0f P b27ba18ec1f24f4f9691a5e7de5ff564 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "936984d765f745738bea4acbbe9c61fb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37753 } } peers { permanent_uuid: "b27ba18ec1f24f4f9691a5e7de5ff564" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37627 } } peers { permanent_uuid: "649983f98f554789af84afcb77b8e38a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46835 } }
I20260929 19:01:21.531610 7919 catalog_manager.cc:5495] T f0349ee8b2d240caa92e259ca18b9d0f P b27ba18ec1f24f4f9691a5e7de5ff564 reported cstate change: term changed from 0 to 1, leader changed from <none> to b27ba18ec1f24f4f9691a5e7de5ff564 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "b27ba18ec1f24f4f9691a5e7de5ff564" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "936984d765f745738bea4acbbe9c61fb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37753 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b27ba18ec1f24f4f9691a5e7de5ff564" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37627 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "649983f98f554789af84afcb77b8e38a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46835 } health_report { overall_health: UNKNOWN } } }
W20260929 19:01:21.562384 8403 tablet.cc:2369] T f0349ee8b2d240caa92e259ca18b9d0f P 649983f98f554789af84afcb77b8e38a: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 29 19:01:21 dist-test-slave-lqkv krb5kdc[7874](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708480, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260929 19:01:21.568588 7968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:21.561534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38778 (local address 127.0.98.62:34221)
0929 19:01:21.561600 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:21.561603 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:21.561719 (+ 116us) server_negotiation.cc:407] Connection header received
0929 19:01:21.561797 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:21.561798 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:21.561810 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:21.561872 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:01:21.562339 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:21.562898 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:21.563603 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:21.563792 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:21.563876 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:21.565982 (+ 2106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:01:21.565984 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:01:21.565984 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:01:21.565991 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:01:21.567407 (+ 1416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:21.567810 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:21.567811 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:21.567811 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:21.567869 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:21.568065 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:21.568066 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:21.568066 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:21.568177 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:01:21.568251 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 19:01:21.568401 (+ 150us) server_negotiation.cc:299] Negotiation successful
0929 19:01:21.568480 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 19:01:21.568668 8467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:21.561490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34221 (local address 127.0.0.1:38778)
0929 19:01:21.561639 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:21.561651 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 19:01:21.561738 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:01:21.561889 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:01:21.561891 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:01:21.562195 (+ 304us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:01:21.562239 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:21.562241 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:21.562980 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:21.562981 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:21.563509 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:21.563511 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:21.563875 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:21.563875 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:21.563923 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:21.564519 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:01:21.564524 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:01:21.565894 (+ 1370us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:01:21.567503 (+ 1609us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:01:21.567509 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:01:21.567512 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:01:21.567735 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:01:21.567939 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:01:21.567940 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:01:21.567941 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:01:21.568003 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:01:21.568243 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:01:21.568249 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:01:21.568294 (+ 45us) client_negotiation.cc:769] Sending connection context
0929 19:01:21.568381 (+ 87us) client_negotiation.cc:240] Negotiation successful
0929 19:01:21.568541 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"mutex_wait_us":65,"thread_start_us":88,"threads_started":1}
I20260929 19:01:21.570701 7919 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38778:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260929 19:01:21.570955 7919 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 19:01:21.572345 8056 tablet_service.cc:1467] Processing CreateTablet for tablet 07f67d6f3df0499eab2f56f47f8df3b1 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=990cabe72a10462eae9818914827fa34]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 19:01:21.572423 8336 tablet_service.cc:1467] Processing CreateTablet for tablet 07f67d6f3df0499eab2f56f47f8df3b1 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=990cabe72a10462eae9818914827fa34]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 19:01:21.572500 8056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07f67d6f3df0499eab2f56f47f8df3b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:01:21.572451 8196 tablet_service.cc:1467] Processing CreateTablet for tablet 07f67d6f3df0499eab2f56f47f8df3b1 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=990cabe72a10462eae9818914827fa34]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 19:01:21.572532 8336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07f67d6f3df0499eab2f56f47f8df3b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:01:21.572595 8196 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07f67d6f3df0499eab2f56f47f8df3b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:01:21.573968 8431 tablet_bootstrap.cc:492] T 07f67d6f3df0499eab2f56f47f8df3b1 P 649983f98f554789af84afcb77b8e38a: Bootstrap starting.
I20260929 19:01:21.574194 8433 tablet_bootstrap.cc:492] T 07f67d6f3df0499eab2f56f47f8df3b1 P b27ba18ec1f24f4f9691a5e7de5ff564: Bootstrap starting.
I20260929 19:01:21.574213 8430 tablet_bootstrap.cc:492] T 07f67d6f3df0499eab2f56f47f8df3b1 P 936984d765f745738bea4acbbe9c61fb: Bootstrap starting.
I20260929 19:01:21.574383 8431 tablet_bootstrap.cc:654] T 07f67d6f3df0499eab2f56f47f8df3b1 P 649983f98f554789af84afcb77b8e38a: Neither blocks nor log segments found. Creating new log.
I20260929 19:01:21.574592 8433 tablet_bootstrap.cc:654] T 07f67d6f3df0499eab2f56f47f8df3b1 P b27ba18ec1f24f4f9691a5e7de5ff564: Neither blocks nor log segments found. Creating new log.
I20260929 19:01:21.574596 8430 tablet_bootstrap.cc:654] T 07f67d6f3df0499eab2f56f47f8df3b1 P 936984d765f745738bea4acbbe9c61fb: Neither blocks nor log segments found. Creating new log.
I20260929 19:01:21.575342 8430 tablet_bootstrap.cc:492] T 07f67d6f3df0499eab2f56f47f8df3b1 P 936984d765f745738bea4acbbe9c61fb: No bootstrap required, opened a new log
I20260929 19:01:21.575393 8430 ts_tablet_manager.cc:1397] T 07f67d6f3df0499eab2f56f47f8df3b1 P 936984d765f745738bea4acbbe9c61fb: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260929 19:01:21.575512 8433 tablet_bootstrap.cc:492] T 07f67d6f3df0499eab2f56f47f8df3b1 P b27ba18ec1f24f4f9691a5e7de5ff564: No bootstrap required, opened a new log
I20260929 19:01:21.575546 8433 ts_tablet_manager.cc:1397] T 07f67d6f3df0499eab2f56f47f8df3b1 P b27ba18ec1f24f4f9691a5e7de5ff564: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260929 19:01:21.575541 8430 raft_consensus.cc:348] T 07f67d6f3df0499eab2f56f47f8df3b1 P 936984d765f745738bea4acbbe9c61fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "936984d765f745738bea4acbbe9c61fb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37753 } } peers { permanent_uuid: "b27ba18ec1f24f4f9691a5e7de5ff564" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37627 } } peers { permanent_uuid: "649983f98f554789af84afcb77b8e38a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46835 } }
I20260929 19:01:21.575605 8430 raft_consensus.cc:374] T 07f67d6f3df0499eab2f56f47f8df3b1 P 936984d765f745738bea4acbbe9c61fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:01:21.575628 8430 raft_consensus.cc:729] T 07f67d6f3df0499eab2f56f47f8df3b1 P 936984d765f745738bea4acbbe9c61fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 936984d765f745738bea4acbbe9c61fb, State: Initialized, Role: FOLLOWER
I20260929 19:01:21.575676 8430 consensus_queue.cc:260] T 07f67d6f3df0499eab2f56f47f8df3b1 P 936984d765f745738bea4acbbe9c61fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "936984d765f745738bea4acbbe9c61fb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37753 } } peers { permanent_uuid: "b27ba18ec1f24f4f9691a5e7de5ff564" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37627 } } peers { permanent_uuid: "649983f98f554789af84afcb77b8e38a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46835 } }
I20260929 19:01:21.575773 8430 ts_tablet_manager.cc:1428] T 07f67d6f3df0499eab2f56f47f8df3b1 P 936984d765f745738bea4acbbe9c61fb: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:21.575852 8433 raft_consensus.cc:348] T 07f67d6f3df0499eab2f56f47f8df3b1 P b27ba18ec1f24f4f9691a5e7de5ff564 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "936984d765f745738bea4acbbe9c61fb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37753 } } peers { permanent_uuid: "b27ba18ec1f24f4f9691a5e7de5ff564" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37627 } } peers { permanent_uuid: "649983f98f554789af84afcb77b8e38a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46835 } }
I20260929 19:01:21.575830 8436 tablet_replica.cc:439] T 07f67d6f3df0499eab2f56f47f8df3b1 P 936984d765f745738bea4acbbe9c61fb: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "936984d765f745738bea4acbbe9c61fb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37753 } } peers { permanent_uuid: "b27ba18ec1f24f4f9691a5e7de5ff564" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37627 } } peers { permanent_uuid: "649983f98f554789af84afcb77b8e38a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46835 } } }
I20260929 19:01:21.575923 8433 raft_consensus.cc:374] T 07f67d6f3df0499eab2f56f47f8df3b1 P b27ba18ec1f24f4f9691a5e7de5ff564 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:01:21.575946 8433 raft_consensus.cc:729] T 07f67d6f3df0499eab2f56f47f8df3b1 P b27ba18ec1f24f4f9691a5e7de5ff564 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b27ba18ec1f24f4f9691a5e7de5ff564, State: Initialized, Role: FOLLOWER
I20260929 19:01:21.575933 8436 tablet_replica.cc:442] T 07f67d6f3df0499eab2f56f47f8df3b1 P 936984d765f745738bea4acbbe9c61fb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 19:01:21.575986 8433 consensus_queue.cc:260] T 07f67d6f3df0499eab2f56f47f8df3b1 P b27ba18ec1f24f4f9691a5e7de5ff564 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "936984d765f745738bea4acbbe9c61fb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37753 } } peers { permanent_uuid: "b27ba18ec1f24f4f9691a5e7de5ff564" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37627 } } peers { permanent_uuid: "649983f98f554789af84afcb77b8e38a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46835 } }
I20260929 19:01:21.576076 8433 ts_tablet_manager.cc:1428] T 07f67d6f3df0499eab2f56f47f8df3b1 P b27ba18ec1f24f4f9691a5e7de5ff564: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260929 19:01:21.576114 8437 tablet_replica.cc:439] T 07f67d6f3df0499eab2f56f47f8df3b1 P b27ba18ec1f24f4f9691a5e7de5ff564: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "936984d765f745738bea4acbbe9c61fb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37753 } } peers { permanent_uuid: "b27ba18ec1f24f4f9691a5e7de5ff564" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37627 } } peers { permanent_uuid: "649983f98f554789af84afcb77b8e38a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46835 } } }
I20260929 19:01:21.576208 8437 tablet_replica.cc:442] T 07f67d6f3df0499eab2f56f47f8df3b1 P b27ba18ec1f24f4f9691a5e7de5ff564: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 19:01:21.576225 8431 tablet_bootstrap.cc:492] T 07f67d6f3df0499eab2f56f47f8df3b1 P 649983f98f554789af84afcb77b8e38a: No bootstrap required, opened a new log
I20260929 19:01:21.576279 8431 ts_tablet_manager.cc:1397] T 07f67d6f3df0499eab2f56f47f8df3b1 P 649983f98f554789af84afcb77b8e38a: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260929 19:01:21.576426 8431 raft_consensus.cc:348] T 07f67d6f3df0499eab2f56f47f8df3b1 P 649983f98f554789af84afcb77b8e38a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "936984d765f745738bea4acbbe9c61fb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37753 } } peers { permanent_uuid: "b27ba18ec1f24f4f9691a5e7de5ff564" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37627 } } peers { permanent_uuid: "649983f98f554789af84afcb77b8e38a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46835 } }
I20260929 19:01:21.576493 8431 raft_consensus.cc:374] T 07f67d6f3df0499eab2f56f47f8df3b1 P 649983f98f554789af84afcb77b8e38a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:01:21.576522 8431 raft_consensus.cc:729] T 07f67d6f3df0499eab2f56f47f8df3b1 P 649983f98f554789af84afcb77b8e38a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 649983f98f554789af84afcb77b8e38a, State: Initialized, Role: FOLLOWER
I20260929 19:01:21.576570 8431 consensus_queue.cc:260] T 07f67d6f3df0499eab2f56f47f8df3b1 P 649983f98f554789af84afcb77b8e38a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "936984d765f745738bea4acbbe9c61fb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37753 } } peers { permanent_uuid: "b27ba18ec1f24f4f9691a5e7de5ff564" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37627 } } peers { permanent_uuid: "649983f98f554789af84afcb77b8e38a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46835 } }
I20260929 19:01:21.576650 8431 ts_tablet_manager.cc:1428] T 07f67d6f3df0499eab2f56f47f8df3b1 P 649983f98f554789af84afcb77b8e38a: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260929 19:01:21.576692 8449 tablet_replica.cc:439] T 07f67d6f3df0499eab2f56f47f8df3b1 P 649983f98f554789af84afcb77b8e38a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "936984d765f745738bea4acbbe9c61fb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37753 } } peers { permanent_uuid: "b27ba18ec1f24f4f9691a5e7de5ff564" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37627 } } peers { permanent_uuid: "649983f98f554789af84afcb77b8e38a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46835 } } }
I20260929 19:01:21.576786 8449 tablet_replica.cc:442] T 07f67d6f3df0499eab2f56f47f8df3b1 P 649983f98f554789af84afcb77b8e38a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 19:01:21.689760 8436 raft_consensus.cc:482] T 07f67d6f3df0499eab2f56f47f8df3b1 P 936984d765f745738bea4acbbe9c61fb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 19:01:21.689882 8436 raft_consensus.cc:504] T 07f67d6f3df0499eab2f56f47f8df3b1 P 936984d765f745738bea4acbbe9c61fb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "936984d765f745738bea4acbbe9c61fb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37753 } } peers { permanent_uuid: "b27ba18ec1f24f4f9691a5e7de5ff564" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37627 } } peers { permanent_uuid: "649983f98f554789af84afcb77b8e38a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46835 } }
I20260929 19:01:21.690286 8436 leader_election.cc:288] T 07f67d6f3df0499eab2f56f47f8df3b1 P 936984d765f745738bea4acbbe9c61fb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b27ba18ec1f24f4f9691a5e7de5ff564 (127.0.98.2:37627), 649983f98f554789af84afcb77b8e38a (127.0.98.3:46835)
I20260929 19:01:21.693333 8429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:21.690552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39529 (local address 127.0.98.2:37627)
0929 19:01:21.690600 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:21.690603 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:21.690804 (+ 201us) server_negotiation.cc:407] Connection header received
0929 19:01:21.690864 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:21.690865 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:21.690874 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:21.690932 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:01:21.691080 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:21.691571 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:21.692517 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:21.692971 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:21.693047 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:21.693059 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0929 19:01:21.693218 (+ 159us) server_negotiation.cc:299] Negotiation successful
0929 19:01:21.693256 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260929 19:01:21.693349 8427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:21.690800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52585 (local address 127.0.98.3:46835)
0929 19:01:21.690879 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:21.690881 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:21.690891 (+ 10us) server_negotiation.cc:407] Connection header received
0929 19:01:21.690916 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:21.690917 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:21.690932 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:21.691006 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:01:21.691184 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:21.691637 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:21.692558 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:21.692998 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:21.693076 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:21.693085 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0929 19:01:21.693247 (+ 162us) server_negotiation.cc:299] Negotiation successful
0929 19:01:21.693280 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 19:01:21.693391 8472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:21.690468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37627 (local address 127.0.98.1:39529)
0929 19:01:21.690747 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:21.690754 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0929 19:01:21.690810 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:01:21.690936 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:01:21.690937 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:01:21.690939 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:01:21.691005 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:21.691007 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:21.691654 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:21.691655 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:21.692422 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:21.692424 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:21.693061 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:21.693062 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:21.693129 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:21.693135 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 19:01:21.693211 (+ 76us) client_negotiation.cc:240] Negotiation successful
0929 19:01:21.693255 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"spinlock_wait_cycles":896,"thread_start_us":84,"threads_started":1}
I20260929 19:01:21.693396 8471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:21.690463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46835 (local address 127.0.98.1:52585)
0929 19:01:21.690624 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:21.690634 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 19:01:21.690695 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:01:21.691012 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:01:21.691014 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:01:21.691016 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:01:21.691104 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:21.691106 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:21.691726 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:21.691727 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:21.692457 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:21.692459 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:21.693076 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:21.693077 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:21.693148 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:21.693151 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 19:01:21.693236 (+ 85us) client_negotiation.cc:240] Negotiation successful
0929 19:01:21.693276 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"mutex_wait_us":1,"thread_start_us":84,"threads_started":1}
I20260929 19:01:21.693755 8356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07f67d6f3df0499eab2f56f47f8df3b1" candidate_uuid: "936984d765f745738bea4acbbe9c61fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "649983f98f554789af84afcb77b8e38a" is_pre_election: true
I20260929 19:01:21.693774 8205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07f67d6f3df0499eab2f56f47f8df3b1" candidate_uuid: "936984d765f745738bea4acbbe9c61fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b27ba18ec1f24f4f9691a5e7de5ff564" is_pre_election: true
I20260929 19:01:21.693846 8356 raft_consensus.cc:2454] T 07f67d6f3df0499eab2f56f47f8df3b1 P 649983f98f554789af84afcb77b8e38a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 936984d765f745738bea4acbbe9c61fb in term 0.
I20260929 19:01:21.693874 8205 raft_consensus.cc:2454] T 07f67d6f3df0499eab2f56f47f8df3b1 P b27ba18ec1f24f4f9691a5e7de5ff564 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 936984d765f745738bea4acbbe9c61fb in term 0.
I20260929 19:01:21.694012 8007 leader_election.cc:302] T 07f67d6f3df0499eab2f56f47f8df3b1 P 936984d765f745738bea4acbbe9c61fb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 649983f98f554789af84afcb77b8e38a, 936984d765f745738bea4acbbe9c61fb; no voters:
I20260929 19:01:21.694180 8436 raft_consensus.cc:2781] T 07f67d6f3df0499eab2f56f47f8df3b1 P 936984d765f745738bea4acbbe9c61fb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 19:01:21.694229 8436 raft_consensus.cc:482] T 07f67d6f3df0499eab2f56f47f8df3b1 P 936984d765f745738bea4acbbe9c61fb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 19:01:21.694244 8436 raft_consensus.cc:3037] T 07f67d6f3df0499eab2f56f47f8df3b1 P 936984d765f745738bea4acbbe9c61fb [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:01:21.694923 8436 raft_consensus.cc:504] T 07f67d6f3df0499eab2f56f47f8df3b1 P 936984d765f745738bea4acbbe9c61fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "936984d765f745738bea4acbbe9c61fb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37753 } } peers { permanent_uuid: "b27ba18ec1f24f4f9691a5e7de5ff564" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37627 } } peers { permanent_uuid: "649983f98f554789af84afcb77b8e38a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46835 } }
I20260929 19:01:21.695055 8436 leader_election.cc:288] T 07f67d6f3df0499eab2f56f47f8df3b1 P 936984d765f745738bea4acbbe9c61fb [CANDIDATE]: Term 1 election: Requested vote from peers b27ba18ec1f24f4f9691a5e7de5ff564 (127.0.98.2:37627), 649983f98f554789af84afcb77b8e38a (127.0.98.3:46835)
I20260929 19:01:21.695185 8356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07f67d6f3df0499eab2f56f47f8df3b1" candidate_uuid: "936984d765f745738bea4acbbe9c61fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "649983f98f554789af84afcb77b8e38a"
I20260929 19:01:21.695264 8356 raft_consensus.cc:3037] T 07f67d6f3df0499eab2f56f47f8df3b1 P 649983f98f554789af84afcb77b8e38a [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:01:21.695284 8205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07f67d6f3df0499eab2f56f47f8df3b1" candidate_uuid: "936984d765f745738bea4acbbe9c61fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b27ba18ec1f24f4f9691a5e7de5ff564"
I20260929 19:01:21.695364 8205 raft_consensus.cc:3037] T 07f67d6f3df0499eab2f56f47f8df3b1 P b27ba18ec1f24f4f9691a5e7de5ff564 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:01:21.695858 8356 raft_consensus.cc:2454] T 07f67d6f3df0499eab2f56f47f8df3b1 P 649983f98f554789af84afcb77b8e38a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 936984d765f745738bea4acbbe9c61fb in term 1.
I20260929 19:01:21.696017 8007 leader_election.cc:302] T 07f67d6f3df0499eab2f56f47f8df3b1 P 936984d765f745738bea4acbbe9c61fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 649983f98f554789af84afcb77b8e38a, 936984d765f745738bea4acbbe9c61fb; no voters:
I20260929 19:01:21.696017 8205 raft_consensus.cc:2454] T 07f67d6f3df0499eab2f56f47f8df3b1 P b27ba18ec1f24f4f9691a5e7de5ff564 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 936984d765f745738bea4acbbe9c61fb in term 1.
I20260929 19:01:21.696115 8436 raft_consensus.cc:2781] T 07f67d6f3df0499eab2f56f47f8df3b1 P 936984d765f745738bea4acbbe9c61fb [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:01:21.696240 8436 raft_consensus.cc:686] T 07f67d6f3df0499eab2f56f47f8df3b1 P 936984d765f745738bea4acbbe9c61fb [term 1 LEADER]: Becoming Leader. State: Replica: 936984d765f745738bea4acbbe9c61fb, State: Running, Role: LEADER
I20260929 19:01:21.696313 8436 consensus_queue.cc:237] T 07f67d6f3df0499eab2f56f47f8df3b1 P 936984d765f745738bea4acbbe9c61fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "936984d765f745738bea4acbbe9c61fb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37753 } } peers { permanent_uuid: "b27ba18ec1f24f4f9691a5e7de5ff564" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37627 } } peers { permanent_uuid: "649983f98f554789af84afcb77b8e38a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46835 } }
I20260929 19:01:21.696671 8473 tablet_replica.cc:439] T 07f67d6f3df0499eab2f56f47f8df3b1 P 936984d765f745738bea4acbbe9c61fb: TxnStatusTablet state changed. Reason: New leader 936984d765f745738bea4acbbe9c61fb. Latest consensus state: current_term: 1 leader_uuid: "936984d765f745738bea4acbbe9c61fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "936984d765f745738bea4acbbe9c61fb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37753 } } peers { permanent_uuid: "b27ba18ec1f24f4f9691a5e7de5ff564" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37627 } } peers { permanent_uuid: "649983f98f554789af84afcb77b8e38a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46835 } } }
I20260929 19:01:21.696787 8473 tablet_replica.cc:442] T 07f67d6f3df0499eab2f56f47f8df3b1 P 936984d765f745738bea4acbbe9c61fb: This TxnStatusTablet replica's current role is: LEADER
I20260929 19:01:21.697022 8475 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260929 19:01:21.697129 8475 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260929 19:01:21.697082 7919 catalog_manager.cc:5495] T 07f67d6f3df0499eab2f56f47f8df3b1 P 936984d765f745738bea4acbbe9c61fb reported cstate change: term changed from 0 to 1, leader changed from <none> to 936984d765f745738bea4acbbe9c61fb (127.0.98.1). New cstate: current_term: 1 leader_uuid: "936984d765f745738bea4acbbe9c61fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "936984d765f745738bea4acbbe9c61fb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37753 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b27ba18ec1f24f4f9691a5e7de5ff564" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37627 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "649983f98f554789af84afcb77b8e38a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46835 } health_report { overall_health: UNKNOWN } } }
I20260929 19:01:21.722885 8467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:21.720319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37753 (local address 127.0.0.1:54918)
0929 19:01:21.720403 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:21.720412 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 19:01:21.720475 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:01:21.720638 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:01:21.720639 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:01:21.720640 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 19:01:21.720693 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:21.720696 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:21.721408 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:21.721408 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:21.721980 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:21.721981 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:21.722351 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:21.722352 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:21.722427 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:21.722452 (+ 25us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 19:01:21.722718 (+ 266us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 19:01:21.722726 (+ 8us) client_negotiation.cc:769] Sending connection context
0929 19:01:21.722765 (+ 39us) client_negotiation.cc:240] Negotiation successful
0929 19:01:21.722819 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260929 19:01:21.722872 8428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:21.720398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54918 (local address 127.0.98.1:37753)
0929 19:01:21.720460 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:21.720462 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:21.720472 (+ 10us) server_negotiation.cc:407] Connection header received
0929 19:01:21.720550 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:21.720551 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:21.720570 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:21.720634 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 19:01:21.720829 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:21.721319 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:21.722085 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:21.722280 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:21.722353 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:21.722547 (+ 194us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 19:01:21.722650 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 19:01:21.722740 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0929 19:01:21.722766 (+ 26us) server_negotiation.cc:299] Negotiation successful
0929 19:01:21.722786 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260929 19:01:21.724076 8205 raft_consensus.cc:1261] T 07f67d6f3df0499eab2f56f47f8df3b1 P b27ba18ec1f24f4f9691a5e7de5ff564 [term 1 FOLLOWER]: Refusing update from remote peer 936984d765f745738bea4acbbe9c61fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 19:01:21.724155 8356 raft_consensus.cc:1261] T 07f67d6f3df0499eab2f56f47f8df3b1 P 649983f98f554789af84afcb77b8e38a [term 1 FOLLOWER]: Refusing update from remote peer 936984d765f745738bea4acbbe9c61fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 19:01:21.724328 8473 consensus_queue.cc:1035] T 07f67d6f3df0499eab2f56f47f8df3b1 P 936984d765f745738bea4acbbe9c61fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "b27ba18ec1f24f4f9691a5e7de5ff564" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37627 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 19:01:21.724433 8436 consensus_queue.cc:1035] T 07f67d6f3df0499eab2f56f47f8df3b1 P 936984d765f745738bea4acbbe9c61fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "649983f98f554789af84afcb77b8e38a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 19:01:21.725023 8437 tablet_replica.cc:439] T 07f67d6f3df0499eab2f56f47f8df3b1 P b27ba18ec1f24f4f9691a5e7de5ff564: TxnStatusTablet state changed. Reason: New leader 936984d765f745738bea4acbbe9c61fb. Latest consensus state: current_term: 1 leader_uuid: "936984d765f745738bea4acbbe9c61fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "936984d765f745738bea4acbbe9c61fb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37753 } } peers { permanent_uuid: "b27ba18ec1f24f4f9691a5e7de5ff564" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37627 } } peers { permanent_uuid: "649983f98f554789af84afcb77b8e38a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46835 } } }
I20260929 19:01:21.725116 8437 tablet_replica.cc:442] T 07f67d6f3df0499eab2f56f47f8df3b1 P b27ba18ec1f24f4f9691a5e7de5ff564: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 19:01:21.725247 8449 tablet_replica.cc:439] T 07f67d6f3df0499eab2f56f47f8df3b1 P 649983f98f554789af84afcb77b8e38a: TxnStatusTablet state changed. Reason: New leader 936984d765f745738bea4acbbe9c61fb. Latest consensus state: current_term: 1 leader_uuid: "936984d765f745738bea4acbbe9c61fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "936984d765f745738bea4acbbe9c61fb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37753 } } peers { permanent_uuid: "b27ba18ec1f24f4f9691a5e7de5ff564" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37627 } } peers { permanent_uuid: "649983f98f554789af84afcb77b8e38a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46835 } } }
I20260929 19:01:21.725327 8449 tablet_replica.cc:442] T 07f67d6f3df0499eab2f56f47f8df3b1 P 649983f98f554789af84afcb77b8e38a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 19:01:21.725860 8436 tablet_replica.cc:439] T 07f67d6f3df0499eab2f56f47f8df3b1 P 936984d765f745738bea4acbbe9c61fb: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "936984d765f745738bea4acbbe9c61fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "936984d765f745738bea4acbbe9c61fb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37753 } } peers { permanent_uuid: "b27ba18ec1f24f4f9691a5e7de5ff564" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37627 } } peers { permanent_uuid: "649983f98f554789af84afcb77b8e38a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46835 } } }
I20260929 19:01:21.725940 8436 tablet_replica.cc:442] T 07f67d6f3df0499eab2f56f47f8df3b1 P 936984d765f745738bea4acbbe9c61fb: This TxnStatusTablet replica's current role is: LEADER
I20260929 19:01:21.726050 8437 tablet_replica.cc:439] T 07f67d6f3df0499eab2f56f47f8df3b1 P b27ba18ec1f24f4f9691a5e7de5ff564: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "936984d765f745738bea4acbbe9c61fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "936984d765f745738bea4acbbe9c61fb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37753 } } peers { permanent_uuid: "b27ba18ec1f24f4f9691a5e7de5ff564" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37627 } } peers { permanent_uuid: "649983f98f554789af84afcb77b8e38a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46835 } } }
I20260929 19:01:21.726130 8437 tablet_replica.cc:442] T 07f67d6f3df0499eab2f56f47f8df3b1 P b27ba18ec1f24f4f9691a5e7de5ff564: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 19:01:21.726755 8436 tablet_replica.cc:439] T 07f67d6f3df0499eab2f56f47f8df3b1 P 936984d765f745738bea4acbbe9c61fb: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "936984d765f745738bea4acbbe9c61fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "936984d765f745738bea4acbbe9c61fb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37753 } } peers { permanent_uuid: "b27ba18ec1f24f4f9691a5e7de5ff564" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37627 } } peers { permanent_uuid: "649983f98f554789af84afcb77b8e38a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46835 } } }
I20260929 19:01:21.726840 8436 tablet_replica.cc:442] T 07f67d6f3df0499eab2f56f47f8df3b1 P 936984d765f745738bea4acbbe9c61fb: This TxnStatusTablet replica's current role is: LEADER
I20260929 19:01:21.727011 8449 tablet_replica.cc:439] T 07f67d6f3df0499eab2f56f47f8df3b1 P 649983f98f554789af84afcb77b8e38a: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "936984d765f745738bea4acbbe9c61fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "936984d765f745738bea4acbbe9c61fb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37753 } } peers { permanent_uuid: "b27ba18ec1f24f4f9691a5e7de5ff564" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37627 } } peers { permanent_uuid: "649983f98f554789af84afcb77b8e38a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46835 } } }
I20260929 19:01:21.727084 8449 tablet_replica.cc:442] T 07f67d6f3df0499eab2f56f47f8df3b1 P 649983f98f554789af84afcb77b8e38a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 19:01:21.730611 8429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:21.728267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60106 (local address 127.0.98.2:37627)
0929 19:01:21.728312 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:21.728313 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:21.728322 (+ 9us) server_negotiation.cc:407] Connection header received
0929 19:01:21.728399 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:21.728400 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:21.728417 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:21.728484 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 19:01:21.728584 (+ 100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:21.729119 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:21.729838 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:21.730035 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:21.730105 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:21.730284 (+ 179us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 19:01:21.730378 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 19:01:21.730456 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 19:01:21.730511 (+ 55us) server_negotiation.cc:299] Negotiation successful
0929 19:01:21.730527 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260929 19:01:21.735532 8428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:21.733063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46367 (local address 127.0.98.1:37753)
0929 19:01:21.733101 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:21.733104 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:21.733234 (+ 130us) server_negotiation.cc:407] Connection header received
0929 19:01:21.733313 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:21.733315 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:21.733330 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:21.733384 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 19:01:21.733524 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:21.733979 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:21.734830 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:21.735037 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:21.735108 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:21.735299 (+ 191us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 19:01:21.735331 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 19:01:21.735395 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0929 19:01:21.735446 (+ 51us) server_negotiation.cc:299] Negotiation successful
0929 19:01:21.735465 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260929 19:01:21.735569 8482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:21.733009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37753 (local address 127.0.98.2:46367)
0929 19:01:21.733154 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:21.733164 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 19:01:21.733249 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:01:21.733387 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:01:21.733387 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:01:21.733389 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 19:01:21.733451 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:21.733452 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:21.734061 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:21.734062 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:21.734740 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:21.734742 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:21.735117 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:21.735119 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:21.735185 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:21.735196 (+ 11us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 19:01:21.735396 (+ 200us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 19:01:21.735399 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 19:01:21.735445 (+ 46us) client_negotiation.cc:240] Negotiation successful
0929 19:01:21.735486 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":76,"threads_started":1}
I20260929 19:01:21.737986 8076 raft_consensus.cc:1261] T f0349ee8b2d240caa92e259ca18b9d0f P 936984d765f745738bea4acbbe9c61fb [term 1 FOLLOWER]: Refusing update from remote peer b27ba18ec1f24f4f9691a5e7de5ff564: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 19:01:21.737986 8356 raft_consensus.cc:1261] T f0349ee8b2d240caa92e259ca18b9d0f P 649983f98f554789af84afcb77b8e38a [term 1 FOLLOWER]: Refusing update from remote peer b27ba18ec1f24f4f9691a5e7de5ff564: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 19:01:21.738233 8460 consensus_queue.cc:1035] T f0349ee8b2d240caa92e259ca18b9d0f P b27ba18ec1f24f4f9691a5e7de5ff564 [LEADER]: Connected to new peer: Peer: permanent_uuid: "649983f98f554789af84afcb77b8e38a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 19:01:21.738348 8437 consensus_queue.cc:1035] T f0349ee8b2d240caa92e259ca18b9d0f P b27ba18ec1f24f4f9691a5e7de5ff564 [LEADER]: Connected to new peer: Peer: permanent_uuid: "936984d765f745738bea4acbbe9c61fb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 19:01:21.747098 8488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:21.744188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37627 (local address 127.0.98.1:51285)
0929 19:01:21.744376 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:21.744388 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 19:01:21.744439 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:01:21.744561 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:01:21.744561 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:01:21.744563 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 19:01:21.744625 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:21.744626 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:21.745314 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:21.745315 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:21.746302 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:21.746304 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:21.746683 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:21.746684 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:21.746745 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:21.746772 (+ 27us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 19:01:21.746968 (+ 196us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 19:01:21.746969 (+ 1us) client_negotiation.cc:769] Sending connection context
0929 19:01:21.747000 (+ 31us) client_negotiation.cc:240] Negotiation successful
0929 19:01:21.747032 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":103,"threads_started":1}
I20260929 19:01:21.747097 8429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:21.744399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51285 (local address 127.0.98.2:37627)
0929 19:01:21.744439 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:21.744441 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:21.744450 (+ 9us) server_negotiation.cc:407] Connection header received
0929 19:01:21.744494 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:21.744495 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:21.744504 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:21.744557 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 19:01:21.744700 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:21.745227 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:21.746378 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:21.746629 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:21.746685 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:21.746870 (+ 185us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 19:01:21.746909 (+ 39us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 19:01:21.746970 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0929 19:01:21.747003 (+ 33us) server_negotiation.cc:299] Negotiation successful
0929 19:01:21.747024 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260929 19:01:21.756100 7917 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38770:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 19:01:21.756217 7917 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38770:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 19:01:21.756788 7917 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1db5b3fcae274bbf94cb5d9e6eff70fa: Sending DeleteTablet for 3 replicas of tablet f0349ee8b2d240caa92e259ca18b9d0f
I20260929 19:01:21.757071 8056 tablet_service.cc:1514] Processing DeleteTablet for tablet f0349ee8b2d240caa92e259ca18b9d0f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:01:21 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54914
I20260929 19:01:21.757071 8196 tablet_service.cc:1514] Processing DeleteTablet for tablet f0349ee8b2d240caa92e259ca18b9d0f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:01:21 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60092
I20260929 19:01:21.757164 8455 tablet_replica.cc:330] T f0349ee8b2d240caa92e259ca18b9d0f P 936984d765f745738bea4acbbe9c61fb: stopping tablet replica
I20260929 19:01:21.757162 8454 tablet_replica.cc:330] T f0349ee8b2d240caa92e259ca18b9d0f P b27ba18ec1f24f4f9691a5e7de5ff564: stopping tablet replica
I20260929 19:01:21.757203 8454 raft_consensus.cc:2229] T f0349ee8b2d240caa92e259ca18b9d0f P b27ba18ec1f24f4f9691a5e7de5ff564 [term 1 LEADER]: Raft consensus shutting down.
I20260929 19:01:21.757203 8455 raft_consensus.cc:2229] T f0349ee8b2d240caa92e259ca18b9d0f P 936984d765f745738bea4acbbe9c61fb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 19:01:21.757237 8455 raft_consensus.cc:2258] T f0349ee8b2d240caa92e259ca18b9d0f P 936984d765f745738bea4acbbe9c61fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 19:01:21.757285 8454 raft_consensus.cc:2258] T f0349ee8b2d240caa92e259ca18b9d0f P b27ba18ec1f24f4f9691a5e7de5ff564 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 19:01:21.757611 8454 ts_tablet_manager.cc:1905] T f0349ee8b2d240caa92e259ca18b9d0f P b27ba18ec1f24f4f9691a5e7de5ff564: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 19:01:21.759104 8455 ts_tablet_manager.cc:1905] T f0349ee8b2d240caa92e259ca18b9d0f P 936984d765f745738bea4acbbe9c61fb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 19:01:21.759198 8336 tablet_service.cc:1514] Processing DeleteTablet for tablet f0349ee8b2d240caa92e259ca18b9d0f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:01:21 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42396
I20260929 19:01:21.759289 8453 tablet_replica.cc:330] T f0349ee8b2d240caa92e259ca18b9d0f P 649983f98f554789af84afcb77b8e38a: stopping tablet replica
I20260929 19:01:21.759337 8453 raft_consensus.cc:2229] T f0349ee8b2d240caa92e259ca18b9d0f P 649983f98f554789af84afcb77b8e38a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 19:01:21.759368 8453 raft_consensus.cc:2258] T f0349ee8b2d240caa92e259ca18b9d0f P 649983f98f554789af84afcb77b8e38a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 19:01:21.761825 8453 ts_tablet_manager.cc:1905] T f0349ee8b2d240caa92e259ca18b9d0f P 649983f98f554789af84afcb77b8e38a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 19:01:21.762521 8454 ts_tablet_manager.cc:1918] T f0349ee8b2d240caa92e259ca18b9d0f P b27ba18ec1f24f4f9691a5e7de5ff564: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 19:01:21.762568 8454 log.cc:1196] T f0349ee8b2d240caa92e259ca18b9d0f P b27ba18ec1f24f4f9691a5e7de5ff564: Deleting WAL directory at /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708447806542-392-0/minicluster-data/ts-1/wal/wals/f0349ee8b2d240caa92e259ca18b9d0f
I20260929 19:01:21.762781 8454 ts_tablet_manager.cc:1939] T f0349ee8b2d240caa92e259ca18b9d0f P b27ba18ec1f24f4f9691a5e7de5ff564: Deleting consensus metadata
I20260929 19:01:21.763072 7906 catalog_manager.cc:4841] TS b27ba18ec1f24f4f9691a5e7de5ff564 (127.0.98.2:37627): tablet f0349ee8b2d240caa92e259ca18b9d0f (table test-table [id=9e35b529267e4e21aab6dfefec8b997c]) successfully deleted
I20260929 19:01:21.763485 8453 ts_tablet_manager.cc:1918] T f0349ee8b2d240caa92e259ca18b9d0f P 649983f98f554789af84afcb77b8e38a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 19:01:21.763535 8453 log.cc:1196] T f0349ee8b2d240caa92e259ca18b9d0f P 649983f98f554789af84afcb77b8e38a: Deleting WAL directory at /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708447806542-392-0/minicluster-data/ts-2/wal/wals/f0349ee8b2d240caa92e259ca18b9d0f
I20260929 19:01:21.763736 8453 ts_tablet_manager.cc:1939] T f0349ee8b2d240caa92e259ca18b9d0f P 649983f98f554789af84afcb77b8e38a: Deleting consensus metadata
I20260929 19:01:21.764047 7903 catalog_manager.cc:4841] TS 649983f98f554789af84afcb77b8e38a (127.0.98.3:46835): tablet f0349ee8b2d240caa92e259ca18b9d0f (table test-table [id=9e35b529267e4e21aab6dfefec8b997c]) successfully deleted
I20260929 19:01:21.764489 8455 ts_tablet_manager.cc:1918] T f0349ee8b2d240caa92e259ca18b9d0f P 936984d765f745738bea4acbbe9c61fb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 19:01:21.764567 8455 log.cc:1196] T f0349ee8b2d240caa92e259ca18b9d0f P 936984d765f745738bea4acbbe9c61fb: Deleting WAL directory at /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708447806542-392-0/minicluster-data/ts-0/wal/wals/f0349ee8b2d240caa92e259ca18b9d0f
I20260929 19:01:21.764760 8455 ts_tablet_manager.cc:1939] T f0349ee8b2d240caa92e259ca18b9d0f P 936984d765f745738bea4acbbe9c61fb: Deleting consensus metadata
I20260929 19:01:21.765097 7905 catalog_manager.cc:4841] TS 936984d765f745738bea4acbbe9c61fb (127.0.98.1:37753): tablet f0349ee8b2d240caa92e259ca18b9d0f (table test-table [id=9e35b529267e4e21aab6dfefec8b997c]) successfully deleted
Sep 29 19:01:21 dist-test-slave-lqkv krb5kdc[7874](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708481, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 19:01:21 dist-test-slave-lqkv krb5kdc[7874](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 19:01:21 dist-test-slave-lqkv krb5kdc[7874](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260929 19:01:21.783957 7968 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:01:21.778571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38786 (local address 127.0.98.62:34221)
0929 19:01:21.778617 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:21.778619 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:21.778758 (+ 139us) server_negotiation.cc:407] Connection header received
0929 19:01:21.778840 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:21.778841 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:21.778856 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:21.778926 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:01:21.779491 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:21.779980 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:21.780731 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:21.781045 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:21.781107 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:21.783885 (+ 2778us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38786: BlockingRecv error: recv got EOF from 127.0.0.1:38786 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
Sep 29 19:01:21 dist-test-slave-lqkv krb5kdc[7874](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708481, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260929 19:01:21.791992 7968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:21.784785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38788 (local address 127.0.98.62:34221)
0929 19:01:21.784880 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:21.784882 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:21.785084 (+ 202us) server_negotiation.cc:407] Connection header received
0929 19:01:21.785145 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:21.785146 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:21.785155 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:21.785226 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:01:21.785734 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:21.786200 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:21.786859 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:21.787052 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:21.787135 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:21.789499 (+ 2364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:01:21.789500 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:01:21.789500 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:01:21.789507 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:01:21.790880 (+ 1373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:21.791233 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:21.791234 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:21.791235 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:21.791299 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:21.791493 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:21.791494 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:21.791495 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:21.791600 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:01:21.791704 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 19:01:21.791812 (+ 108us) server_negotiation.cc:299] Negotiation successful
0929 19:01:21.791883 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260929 19:01:21.793555 7917 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38788:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 19:01:21.793727 7917 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 19:01:21.795030 8196 tablet_service.cc:1467] Processing CreateTablet for tablet 824d06886ba141bb8efe08abf7a43c17 (DEFAULT_TABLE table=test-table [id=899412e73d26478daea02a0f09d2eb5d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:01:21.795030 8056 tablet_service.cc:1467] Processing CreateTablet for tablet 824d06886ba141bb8efe08abf7a43c17 (DEFAULT_TABLE table=test-table [id=899412e73d26478daea02a0f09d2eb5d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:01:21.795156 8196 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 824d06886ba141bb8efe08abf7a43c17. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:01:21.795156 8056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 824d06886ba141bb8efe08abf7a43c17. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:01:21.796432 8433 tablet_bootstrap.cc:492] T 824d06886ba141bb8efe08abf7a43c17 P b27ba18ec1f24f4f9691a5e7de5ff564: Bootstrap starting.
I20260929 19:01:21.796432 8430 tablet_bootstrap.cc:492] T 824d06886ba141bb8efe08abf7a43c17 P 936984d765f745738bea4acbbe9c61fb: Bootstrap starting.
I20260929 19:01:21.796873 8433 tablet_bootstrap.cc:654] T 824d06886ba141bb8efe08abf7a43c17 P b27ba18ec1f24f4f9691a5e7de5ff564: Neither blocks nor log segments found. Creating new log.
I20260929 19:01:21.797320 8430 tablet_bootstrap.cc:654] T 824d06886ba141bb8efe08abf7a43c17 P 936984d765f745738bea4acbbe9c61fb: Neither blocks nor log segments found. Creating new log.
I20260929 19:01:21.797313 8336 tablet_service.cc:1467] Processing CreateTablet for tablet 824d06886ba141bb8efe08abf7a43c17 (DEFAULT_TABLE table=test-table [id=899412e73d26478daea02a0f09d2eb5d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:01:21.797441 8336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 824d06886ba141bb8efe08abf7a43c17. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:01:21.797947 8433 tablet_bootstrap.cc:492] T 824d06886ba141bb8efe08abf7a43c17 P b27ba18ec1f24f4f9691a5e7de5ff564: No bootstrap required, opened a new log
I20260929 19:01:21.797993 8433 ts_tablet_manager.cc:1397] T 824d06886ba141bb8efe08abf7a43c17 P b27ba18ec1f24f4f9691a5e7de5ff564: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260929 19:01:21.798146 8433 raft_consensus.cc:348] T 824d06886ba141bb8efe08abf7a43c17 P b27ba18ec1f24f4f9691a5e7de5ff564 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "936984d765f745738bea4acbbe9c61fb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37753 } } peers { permanent_uuid: "b27ba18ec1f24f4f9691a5e7de5ff564" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37627 } } peers { permanent_uuid: "649983f98f554789af84afcb77b8e38a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46835 } }
I20260929 19:01:21.798218 8433 raft_consensus.cc:374] T 824d06886ba141bb8efe08abf7a43c17 P b27ba18ec1f24f4f9691a5e7de5ff564 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:01:21.798239 8433 raft_consensus.cc:729] T 824d06886ba141bb8efe08abf7a43c17 P b27ba18ec1f24f4f9691a5e7de5ff564 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b27ba18ec1f24f4f9691a5e7de5ff564, State: Initialized, Role: FOLLOWER
I20260929 19:01:21.798300 8433 consensus_queue.cc:260] T 824d06886ba141bb8efe08abf7a43c17 P b27ba18ec1f24f4f9691a5e7de5ff564 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "936984d765f745738bea4acbbe9c61fb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37753 } } peers { permanent_uuid: "b27ba18ec1f24f4f9691a5e7de5ff564" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37627 } } peers { permanent_uuid: "649983f98f554789af84afcb77b8e38a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46835 } }
I20260929 19:01:21.798391 8433 ts_tablet_manager.cc:1428] T 824d06886ba141bb8efe08abf7a43c17 P b27ba18ec1f24f4f9691a5e7de5ff564: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:21.798805 8431 tablet_bootstrap.cc:492] T 824d06886ba141bb8efe08abf7a43c17 P 649983f98f554789af84afcb77b8e38a: Bootstrap starting.
I20260929 19:01:21.799144 8431 tablet_bootstrap.cc:654] T 824d06886ba141bb8efe08abf7a43c17 P 649983f98f554789af84afcb77b8e38a: Neither blocks nor log segments found. Creating new log.
I20260929 19:01:21.799643 8431 tablet_bootstrap.cc:492] T 824d06886ba141bb8efe08abf7a43c17 P 649983f98f554789af84afcb77b8e38a: No bootstrap required, opened a new log
I20260929 19:01:21.799690 8431 ts_tablet_manager.cc:1397] T 824d06886ba141bb8efe08abf7a43c17 P 649983f98f554789af84afcb77b8e38a: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260929 19:01:21.799793 8431 raft_consensus.cc:348] T 824d06886ba141bb8efe08abf7a43c17 P 649983f98f554789af84afcb77b8e38a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "936984d765f745738bea4acbbe9c61fb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37753 } } peers { permanent_uuid: "b27ba18ec1f24f4f9691a5e7de5ff564" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37627 } } peers { permanent_uuid: "649983f98f554789af84afcb77b8e38a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46835 } }
I20260929 19:01:21.799865 8431 raft_consensus.cc:374] T 824d06886ba141bb8efe08abf7a43c17 P 649983f98f554789af84afcb77b8e38a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:01:21.799885 8431 raft_consensus.cc:729] T 824d06886ba141bb8efe08abf7a43c17 P 649983f98f554789af84afcb77b8e38a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 649983f98f554789af84afcb77b8e38a, State: Initialized, Role: FOLLOWER
I20260929 19:01:21.799927 8431 consensus_queue.cc:260] T 824d06886ba141bb8efe08abf7a43c17 P 649983f98f554789af84afcb77b8e38a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "936984d765f745738bea4acbbe9c61fb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37753 } } peers { permanent_uuid: "b27ba18ec1f24f4f9691a5e7de5ff564" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37627 } } peers { permanent_uuid: "649983f98f554789af84afcb77b8e38a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46835 } }
I20260929 19:01:21.800016 8431 ts_tablet_manager.cc:1428] T 824d06886ba141bb8efe08abf7a43c17 P 649983f98f554789af84afcb77b8e38a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:21.800256 8430 tablet_bootstrap.cc:492] T 824d06886ba141bb8efe08abf7a43c17 P 936984d765f745738bea4acbbe9c61fb: No bootstrap required, opened a new log
I20260929 19:01:21.800314 8430 ts_tablet_manager.cc:1397] T 824d06886ba141bb8efe08abf7a43c17 P 936984d765f745738bea4acbbe9c61fb: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20260929 19:01:21.800463 8430 raft_consensus.cc:348] T 824d06886ba141bb8efe08abf7a43c17 P 936984d765f745738bea4acbbe9c61fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "936984d765f745738bea4acbbe9c61fb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37753 } } peers { permanent_uuid: "b27ba18ec1f24f4f9691a5e7de5ff564" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37627 } } peers { permanent_uuid: "649983f98f554789af84afcb77b8e38a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46835 } }
I20260929 19:01:21.800534 8430 raft_consensus.cc:374] T 824d06886ba141bb8efe08abf7a43c17 P 936984d765f745738bea4acbbe9c61fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:01:21.800554 8430 raft_consensus.cc:729] T 824d06886ba141bb8efe08abf7a43c17 P 936984d765f745738bea4acbbe9c61fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 936984d765f745738bea4acbbe9c61fb, State: Initialized, Role: FOLLOWER
I20260929 19:01:21.800591 8430 consensus_queue.cc:260] T 824d06886ba141bb8efe08abf7a43c17 P 936984d765f745738bea4acbbe9c61fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "936984d765f745738bea4acbbe9c61fb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37753 } } peers { permanent_uuid: "b27ba18ec1f24f4f9691a5e7de5ff564" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37627 } } peers { permanent_uuid: "649983f98f554789af84afcb77b8e38a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46835 } }
I20260929 19:01:21.800691 8430 ts_tablet_manager.cc:1428] T 824d06886ba141bb8efe08abf7a43c17 P 936984d765f745738bea4acbbe9c61fb: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:22.057704 8449 raft_consensus.cc:482] T 824d06886ba141bb8efe08abf7a43c17 P 649983f98f554789af84afcb77b8e38a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 19:01:22.057813 8449 raft_consensus.cc:504] T 824d06886ba141bb8efe08abf7a43c17 P 649983f98f554789af84afcb77b8e38a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "936984d765f745738bea4acbbe9c61fb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37753 } } peers { permanent_uuid: "b27ba18ec1f24f4f9691a5e7de5ff564" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37627 } } peers { permanent_uuid: "649983f98f554789af84afcb77b8e38a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46835 } }
I20260929 19:01:22.058015 8449 leader_election.cc:288] T 824d06886ba141bb8efe08abf7a43c17 P 649983f98f554789af84afcb77b8e38a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 936984d765f745738bea4acbbe9c61fb (127.0.98.1:37753), b27ba18ec1f24f4f9691a5e7de5ff564 (127.0.98.2:37627)
I20260929 19:01:22.058293 8076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "824d06886ba141bb8efe08abf7a43c17" candidate_uuid: "649983f98f554789af84afcb77b8e38a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "936984d765f745738bea4acbbe9c61fb" is_pre_election: true
I20260929 19:01:22.058328 8205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "824d06886ba141bb8efe08abf7a43c17" candidate_uuid: "649983f98f554789af84afcb77b8e38a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b27ba18ec1f24f4f9691a5e7de5ff564" is_pre_election: true
I20260929 19:01:22.058409 8076 raft_consensus.cc:2454] T 824d06886ba141bb8efe08abf7a43c17 P 936984d765f745738bea4acbbe9c61fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 649983f98f554789af84afcb77b8e38a in term 0.
I20260929 19:01:22.058439 8205 raft_consensus.cc:2454] T 824d06886ba141bb8efe08abf7a43c17 P b27ba18ec1f24f4f9691a5e7de5ff564 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 649983f98f554789af84afcb77b8e38a in term 0.
I20260929 19:01:22.058568 8289 leader_election.cc:302] T 824d06886ba141bb8efe08abf7a43c17 P 649983f98f554789af84afcb77b8e38a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 649983f98f554789af84afcb77b8e38a, 936984d765f745738bea4acbbe9c61fb; no voters:
I20260929 19:01:22.058665 8449 raft_consensus.cc:2781] T 824d06886ba141bb8efe08abf7a43c17 P 649983f98f554789af84afcb77b8e38a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 19:01:22.058714 8449 raft_consensus.cc:482] T 824d06886ba141bb8efe08abf7a43c17 P 649983f98f554789af84afcb77b8e38a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 19:01:22.058738 8449 raft_consensus.cc:3037] T 824d06886ba141bb8efe08abf7a43c17 P 649983f98f554789af84afcb77b8e38a [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:01:22.059545 8449 raft_consensus.cc:504] T 824d06886ba141bb8efe08abf7a43c17 P 649983f98f554789af84afcb77b8e38a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "936984d765f745738bea4acbbe9c61fb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37753 } } peers { permanent_uuid: "b27ba18ec1f24f4f9691a5e7de5ff564" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37627 } } peers { permanent_uuid: "649983f98f554789af84afcb77b8e38a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46835 } }
I20260929 19:01:22.059684 8449 leader_election.cc:288] T 824d06886ba141bb8efe08abf7a43c17 P 649983f98f554789af84afcb77b8e38a [CANDIDATE]: Term 1 election: Requested vote from peers 936984d765f745738bea4acbbe9c61fb (127.0.98.1:37753), b27ba18ec1f24f4f9691a5e7de5ff564 (127.0.98.2:37627)
I20260929 19:01:22.059851 8076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "824d06886ba141bb8efe08abf7a43c17" candidate_uuid: "649983f98f554789af84afcb77b8e38a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "936984d765f745738bea4acbbe9c61fb"
I20260929 19:01:22.059868 8205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "824d06886ba141bb8efe08abf7a43c17" candidate_uuid: "649983f98f554789af84afcb77b8e38a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b27ba18ec1f24f4f9691a5e7de5ff564"
I20260929 19:01:22.059944 8076 raft_consensus.cc:3037] T 824d06886ba141bb8efe08abf7a43c17 P 936984d765f745738bea4acbbe9c61fb [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:01:22.059942 8205 raft_consensus.cc:3037] T 824d06886ba141bb8efe08abf7a43c17 P b27ba18ec1f24f4f9691a5e7de5ff564 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:01:22.060657 8076 raft_consensus.cc:2454] T 824d06886ba141bb8efe08abf7a43c17 P 936984d765f745738bea4acbbe9c61fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 649983f98f554789af84afcb77b8e38a in term 1.
I20260929 19:01:22.060676 8205 raft_consensus.cc:2454] T 824d06886ba141bb8efe08abf7a43c17 P b27ba18ec1f24f4f9691a5e7de5ff564 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 649983f98f554789af84afcb77b8e38a in term 1.
I20260929 19:01:22.060855 8289 leader_election.cc:302] T 824d06886ba141bb8efe08abf7a43c17 P 649983f98f554789af84afcb77b8e38a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 649983f98f554789af84afcb77b8e38a, 936984d765f745738bea4acbbe9c61fb; no voters:
I20260929 19:01:22.060961 8449 raft_consensus.cc:2781] T 824d06886ba141bb8efe08abf7a43c17 P 649983f98f554789af84afcb77b8e38a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:01:22.061094 8449 raft_consensus.cc:686] T 824d06886ba141bb8efe08abf7a43c17 P 649983f98f554789af84afcb77b8e38a [term 1 LEADER]: Becoming Leader. State: Replica: 649983f98f554789af84afcb77b8e38a, State: Running, Role: LEADER
I20260929 19:01:22.061183 8449 consensus_queue.cc:237] T 824d06886ba141bb8efe08abf7a43c17 P 649983f98f554789af84afcb77b8e38a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "936984d765f745738bea4acbbe9c61fb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37753 } } peers { permanent_uuid: "b27ba18ec1f24f4f9691a5e7de5ff564" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37627 } } peers { permanent_uuid: "649983f98f554789af84afcb77b8e38a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46835 } }
I20260929 19:01:22.061695 7917 catalog_manager.cc:5495] T 824d06886ba141bb8efe08abf7a43c17 P 649983f98f554789af84afcb77b8e38a reported cstate change: term changed from 0 to 1, leader changed from <none> to 649983f98f554789af84afcb77b8e38a (127.0.98.3). New cstate: current_term: 1 leader_uuid: "649983f98f554789af84afcb77b8e38a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "936984d765f745738bea4acbbe9c61fb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37753 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b27ba18ec1f24f4f9691a5e7de5ff564" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37627 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "649983f98f554789af84afcb77b8e38a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46835 } health_report { overall_health: HEALTHY } } }
I20260929 19:01:22.085031 8427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:22.082811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42420 (local address 127.0.98.3:46835)
0929 19:01:22.082882 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:22.082884 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:22.082899 (+ 15us) server_negotiation.cc:407] Connection header received
0929 19:01:22.082920 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:22.082921 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:22.082930 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:22.082985 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 19:01:22.083083 (+ 98us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:22.083579 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:22.084409 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:22.084592 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:22.084645 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:22.084773 (+ 128us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 19:01:22.084797 (+ 24us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 19:01:22.084893 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0929 19:01:22.084959 (+ 66us) server_negotiation.cc:299] Negotiation successful
0929 19:01:22.084977 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260929 19:01:22.085755 8076 raft_consensus.cc:1261] T 824d06886ba141bb8efe08abf7a43c17 P 936984d765f745738bea4acbbe9c61fb [term 1 FOLLOWER]: Refusing update from remote peer 649983f98f554789af84afcb77b8e38a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 19:01:22.085755 8205 raft_consensus.cc:1261] T 824d06886ba141bb8efe08abf7a43c17 P b27ba18ec1f24f4f9691a5e7de5ff564 [term 1 FOLLOWER]: Refusing update from remote peer 649983f98f554789af84afcb77b8e38a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 19:01:22.085981 8449 consensus_queue.cc:1035] T 824d06886ba141bb8efe08abf7a43c17 P 649983f98f554789af84afcb77b8e38a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b27ba18ec1f24f4f9691a5e7de5ff564" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37627 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 19:01:22.086067 8506 consensus_queue.cc:1035] T 824d06886ba141bb8efe08abf7a43c17 P 649983f98f554789af84afcb77b8e38a [LEADER]: Connected to new peer: Peer: permanent_uuid: "936984d765f745738bea4acbbe9c61fb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 19:01:22.087141 8443 mvcc.cc:205] Tried to move back new op lower bound from 7334741942620889088 to 7334741942522347520. Current Snapshot: MvccSnapshot[applied={T|T < 7334741942620889088}]
I20260929 19:01:22.088532 7919 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38788:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 19:01:22.088603 7919 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38788:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 19:01:22.089205 7919 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1db5b3fcae274bbf94cb5d9e6eff70fa: Sending DeleteTablet for 3 replicas of tablet 824d06886ba141bb8efe08abf7a43c17
I20260929 19:01:22.089416 8056 tablet_service.cc:1514] Processing DeleteTablet for tablet 824d06886ba141bb8efe08abf7a43c17 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:01:22 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54914
I20260929 19:01:22.089437 8196 tablet_service.cc:1514] Processing DeleteTablet for tablet 824d06886ba141bb8efe08abf7a43c17 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:01:22 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60092
I20260929 19:01:22.089437 8336 tablet_service.cc:1514] Processing DeleteTablet for tablet 824d06886ba141bb8efe08abf7a43c17 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:01:22 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42396
I20260929 19:01:22.089515 8454 tablet_replica.cc:330] T 824d06886ba141bb8efe08abf7a43c17 P b27ba18ec1f24f4f9691a5e7de5ff564: stopping tablet replica
I20260929 19:01:22.089525 8453 tablet_replica.cc:330] T 824d06886ba141bb8efe08abf7a43c17 P 649983f98f554789af84afcb77b8e38a: stopping tablet replica
I20260929 19:01:22.089531 8455 tablet_replica.cc:330] T 824d06886ba141bb8efe08abf7a43c17 P 936984d765f745738bea4acbbe9c61fb: stopping tablet replica
I20260929 19:01:22.089567 8453 raft_consensus.cc:2229] T 824d06886ba141bb8efe08abf7a43c17 P 649983f98f554789af84afcb77b8e38a [term 1 LEADER]: Raft consensus shutting down.
I20260929 19:01:22.089569 8454 raft_consensus.cc:2229] T 824d06886ba141bb8efe08abf7a43c17 P b27ba18ec1f24f4f9691a5e7de5ff564 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 19:01:22.089572 8455 raft_consensus.cc:2229] T 824d06886ba141bb8efe08abf7a43c17 P 936984d765f745738bea4acbbe9c61fb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 19:01:22.089607 8454 raft_consensus.cc:2258] T 824d06886ba141bb8efe08abf7a43c17 P b27ba18ec1f24f4f9691a5e7de5ff564 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 19:01:22.089619 8455 raft_consensus.cc:2258] T 824d06886ba141bb8efe08abf7a43c17 P 936984d765f745738bea4acbbe9c61fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 19:01:22.089615 8453 raft_consensus.cc:2258] T 824d06886ba141bb8efe08abf7a43c17 P 649983f98f554789af84afcb77b8e38a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 19:01:22.089867 8455 ts_tablet_manager.cc:1905] T 824d06886ba141bb8efe08abf7a43c17 P 936984d765f745738bea4acbbe9c61fb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 19:01:22.089988 8454 ts_tablet_manager.cc:1905] T 824d06886ba141bb8efe08abf7a43c17 P b27ba18ec1f24f4f9691a5e7de5ff564: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 19:01:22.090093 7919 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38788:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 19:01:22.090216 7919 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 19:01:22.091343 8455 ts_tablet_manager.cc:1918] T 824d06886ba141bb8efe08abf7a43c17 P 936984d765f745738bea4acbbe9c61fb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 19:01:22.091343 8454 ts_tablet_manager.cc:1918] T 824d06886ba141bb8efe08abf7a43c17 P b27ba18ec1f24f4f9691a5e7de5ff564: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 19:01:22.091349 8453 ts_tablet_manager.cc:1905] T 824d06886ba141bb8efe08abf7a43c17 P 649983f98f554789af84afcb77b8e38a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 19:01:22.091400 8455 log.cc:1196] T 824d06886ba141bb8efe08abf7a43c17 P 936984d765f745738bea4acbbe9c61fb: Deleting WAL directory at /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708447806542-392-0/minicluster-data/ts-0/wal/wals/824d06886ba141bb8efe08abf7a43c17
I20260929 19:01:22.091400 8454 log.cc:1196] T 824d06886ba141bb8efe08abf7a43c17 P b27ba18ec1f24f4f9691a5e7de5ff564: Deleting WAL directory at /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708447806542-392-0/minicluster-data/ts-1/wal/wals/824d06886ba141bb8efe08abf7a43c17
I20260929 19:01:22.091638 8454 ts_tablet_manager.cc:1939] T 824d06886ba141bb8efe08abf7a43c17 P b27ba18ec1f24f4f9691a5e7de5ff564: Deleting consensus metadata
I20260929 19:01:22.091638 8455 ts_tablet_manager.cc:1939] T 824d06886ba141bb8efe08abf7a43c17 P 936984d765f745738bea4acbbe9c61fb: Deleting consensus metadata
I20260929 19:01:22.091943 7905 catalog_manager.cc:4841] TS 936984d765f745738bea4acbbe9c61fb (127.0.98.1:37753): tablet 824d06886ba141bb8efe08abf7a43c17 (table test-table [id=899412e73d26478daea02a0f09d2eb5d]) successfully deleted
I20260929 19:01:22.091964 7906 catalog_manager.cc:4841] TS b27ba18ec1f24f4f9691a5e7de5ff564 (127.0.98.2:37627): tablet 824d06886ba141bb8efe08abf7a43c17 (table test-table [id=899412e73d26478daea02a0f09d2eb5d]) successfully deleted
I20260929 19:01:22.092641 8056 tablet_service.cc:1467] Processing CreateTablet for tablet 9aec224873654d2ca51ece90012f8ecb (DEFAULT_TABLE table=test-table [id=0388bd084fb44761b49564b16e15bc35]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:01:22.092670 8336 tablet_service.cc:1467] Processing CreateTablet for tablet 9aec224873654d2ca51ece90012f8ecb (DEFAULT_TABLE table=test-table [id=0388bd084fb44761b49564b16e15bc35]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:01:22.092691 8453 ts_tablet_manager.cc:1918] T 824d06886ba141bb8efe08abf7a43c17 P 649983f98f554789af84afcb77b8e38a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 19:01:22.092739 8453 log.cc:1196] T 824d06886ba141bb8efe08abf7a43c17 P 649983f98f554789af84afcb77b8e38a: Deleting WAL directory at /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708447806542-392-0/minicluster-data/ts-2/wal/wals/824d06886ba141bb8efe08abf7a43c17
I20260929 19:01:22.092751 8056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9aec224873654d2ca51ece90012f8ecb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:01:22.092765 8336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9aec224873654d2ca51ece90012f8ecb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:01:22.092973 8453 ts_tablet_manager.cc:1939] T 824d06886ba141bb8efe08abf7a43c17 P 649983f98f554789af84afcb77b8e38a: Deleting consensus metadata
I20260929 19:01:22.093014 8196 tablet_service.cc:1467] Processing CreateTablet for tablet 9aec224873654d2ca51ece90012f8ecb (DEFAULT_TABLE table=test-table [id=0388bd084fb44761b49564b16e15bc35]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:01:22.093104 8196 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9aec224873654d2ca51ece90012f8ecb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:01:22.093982 8431 tablet_bootstrap.cc:492] T 9aec224873654d2ca51ece90012f8ecb P 649983f98f554789af84afcb77b8e38a: Bootstrap starting.
I20260929 19:01:22.094301 8431 tablet_bootstrap.cc:654] T 9aec224873654d2ca51ece90012f8ecb P 649983f98f554789af84afcb77b8e38a: Neither blocks nor log segments found. Creating new log.
I20260929 19:01:22.094314 8433 tablet_bootstrap.cc:492] T 9aec224873654d2ca51ece90012f8ecb P b27ba18ec1f24f4f9691a5e7de5ff564: Bootstrap starting.
I20260929 19:01:22.094333 8430 tablet_bootstrap.cc:492] T 9aec224873654d2ca51ece90012f8ecb P 936984d765f745738bea4acbbe9c61fb: Bootstrap starting.
I20260929 19:01:22.094611 8433 tablet_bootstrap.cc:654] T 9aec224873654d2ca51ece90012f8ecb P b27ba18ec1f24f4f9691a5e7de5ff564: Neither blocks nor log segments found. Creating new log.
I20260929 19:01:22.094619 8430 tablet_bootstrap.cc:654] T 9aec224873654d2ca51ece90012f8ecb P 936984d765f745738bea4acbbe9c61fb: Neither blocks nor log segments found. Creating new log.
I20260929 19:01:22.094820 8431 tablet_bootstrap.cc:492] T 9aec224873654d2ca51ece90012f8ecb P 649983f98f554789af84afcb77b8e38a: No bootstrap required, opened a new log
I20260929 19:01:22.094861 8431 ts_tablet_manager.cc:1397] T 9aec224873654d2ca51ece90012f8ecb P 649983f98f554789af84afcb77b8e38a: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260929 19:01:22.094926 7903 catalog_manager.cc:4841] TS 649983f98f554789af84afcb77b8e38a (127.0.98.3:46835): tablet 824d06886ba141bb8efe08abf7a43c17 (table test-table [id=899412e73d26478daea02a0f09d2eb5d]) successfully deleted
I20260929 19:01:22.094970 8431 raft_consensus.cc:348] T 9aec224873654d2ca51ece90012f8ecb P 649983f98f554789af84afcb77b8e38a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "936984d765f745738bea4acbbe9c61fb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37753 } } peers { permanent_uuid: "b27ba18ec1f24f4f9691a5e7de5ff564" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37627 } } peers { permanent_uuid: "649983f98f554789af84afcb77b8e38a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46835 } }
I20260929 19:01:22.095032 8431 raft_consensus.cc:374] T 9aec224873654d2ca51ece90012f8ecb P 649983f98f554789af84afcb77b8e38a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:01:22.095057 8431 raft_consensus.cc:729] T 9aec224873654d2ca51ece90012f8ecb P 649983f98f554789af84afcb77b8e38a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 649983f98f554789af84afcb77b8e38a, State: Initialized, Role: FOLLOWER
I20260929 19:01:22.095103 8431 consensus_queue.cc:260] T 9aec224873654d2ca51ece90012f8ecb P 649983f98f554789af84afcb77b8e38a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "936984d765f745738bea4acbbe9c61fb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37753 } } peers { permanent_uuid: "b27ba18ec1f24f4f9691a5e7de5ff564" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37627 } } peers { permanent_uuid: "649983f98f554789af84afcb77b8e38a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46835 } }
I20260929 19:01:22.095152 8430 tablet_bootstrap.cc:492] T 9aec224873654d2ca51ece90012f8ecb P 936984d765f745738bea4acbbe9c61fb: No bootstrap required, opened a new log
I20260929 19:01:22.095186 8430 ts_tablet_manager.cc:1397] T 9aec224873654d2ca51ece90012f8ecb P 936984d765f745738bea4acbbe9c61fb: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260929 19:01:22.095189 8431 ts_tablet_manager.cc:1428] T 9aec224873654d2ca51ece90012f8ecb P 649983f98f554789af84afcb77b8e38a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:22.095319 8430 raft_consensus.cc:348] T 9aec224873654d2ca51ece90012f8ecb P 936984d765f745738bea4acbbe9c61fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "936984d765f745738bea4acbbe9c61fb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37753 } } peers { permanent_uuid: "b27ba18ec1f24f4f9691a5e7de5ff564" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37627 } } peers { permanent_uuid: "649983f98f554789af84afcb77b8e38a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46835 } }
I20260929 19:01:22.095383 8430 raft_consensus.cc:374] T 9aec224873654d2ca51ece90012f8ecb P 936984d765f745738bea4acbbe9c61fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:01:22.095408 8430 raft_consensus.cc:729] T 9aec224873654d2ca51ece90012f8ecb P 936984d765f745738bea4acbbe9c61fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 936984d765f745738bea4acbbe9c61fb, State: Initialized, Role: FOLLOWER
I20260929 19:01:22.095449 8430 consensus_queue.cc:260] T 9aec224873654d2ca51ece90012f8ecb P 936984d765f745738bea4acbbe9c61fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "936984d765f745738bea4acbbe9c61fb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37753 } } peers { permanent_uuid: "b27ba18ec1f24f4f9691a5e7de5ff564" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37627 } } peers { permanent_uuid: "649983f98f554789af84afcb77b8e38a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46835 } }
I20260929 19:01:22.095561 8430 ts_tablet_manager.cc:1428] T 9aec224873654d2ca51ece90012f8ecb P 936984d765f745738bea4acbbe9c61fb: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:22.095811 8433 tablet_bootstrap.cc:492] T 9aec224873654d2ca51ece90012f8ecb P b27ba18ec1f24f4f9691a5e7de5ff564: No bootstrap required, opened a new log
I20260929 19:01:22.095861 8433 ts_tablet_manager.cc:1397] T 9aec224873654d2ca51ece90012f8ecb P b27ba18ec1f24f4f9691a5e7de5ff564: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260929 19:01:22.095971 8433 raft_consensus.cc:348] T 9aec224873654d2ca51ece90012f8ecb P b27ba18ec1f24f4f9691a5e7de5ff564 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "936984d765f745738bea4acbbe9c61fb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37753 } } peers { permanent_uuid: "b27ba18ec1f24f4f9691a5e7de5ff564" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37627 } } peers { permanent_uuid: "649983f98f554789af84afcb77b8e38a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46835 } }
I20260929 19:01:22.096041 8433 raft_consensus.cc:374] T 9aec224873654d2ca51ece90012f8ecb P b27ba18ec1f24f4f9691a5e7de5ff564 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:01:22.096065 8433 raft_consensus.cc:729] T 9aec224873654d2ca51ece90012f8ecb P b27ba18ec1f24f4f9691a5e7de5ff564 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b27ba18ec1f24f4f9691a5e7de5ff564, State: Initialized, Role: FOLLOWER
I20260929 19:01:22.096108 8433 consensus_queue.cc:260] T 9aec224873654d2ca51ece90012f8ecb P b27ba18ec1f24f4f9691a5e7de5ff564 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "936984d765f745738bea4acbbe9c61fb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37753 } } peers { permanent_uuid: "b27ba18ec1f24f4f9691a5e7de5ff564" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37627 } } peers { permanent_uuid: "649983f98f554789af84afcb77b8e38a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46835 } }
I20260929 19:01:22.096200 8433 ts_tablet_manager.cc:1428] T 9aec224873654d2ca51ece90012f8ecb P b27ba18ec1f24f4f9691a5e7de5ff564: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:22.198642 8506 raft_consensus.cc:482] T 9aec224873654d2ca51ece90012f8ecb P 649983f98f554789af84afcb77b8e38a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 19:01:22.198743 8506 raft_consensus.cc:504] T 9aec224873654d2ca51ece90012f8ecb P 649983f98f554789af84afcb77b8e38a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "936984d765f745738bea4acbbe9c61fb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37753 } } peers { permanent_uuid: "b27ba18ec1f24f4f9691a5e7de5ff564" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37627 } } peers { permanent_uuid: "649983f98f554789af84afcb77b8e38a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46835 } }
I20260929 19:01:22.198951 8506 leader_election.cc:288] T 9aec224873654d2ca51ece90012f8ecb P 649983f98f554789af84afcb77b8e38a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 936984d765f745738bea4acbbe9c61fb (127.0.98.1:37753), b27ba18ec1f24f4f9691a5e7de5ff564 (127.0.98.2:37627)
I20260929 19:01:22.199132 8076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9aec224873654d2ca51ece90012f8ecb" candidate_uuid: "649983f98f554789af84afcb77b8e38a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "936984d765f745738bea4acbbe9c61fb" is_pre_election: true
I20260929 19:01:22.199213 8076 raft_consensus.cc:2454] T 9aec224873654d2ca51ece90012f8ecb P 936984d765f745738bea4acbbe9c61fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 649983f98f554789af84afcb77b8e38a in term 0.
I20260929 19:01:22.199186 8205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9aec224873654d2ca51ece90012f8ecb" candidate_uuid: "649983f98f554789af84afcb77b8e38a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b27ba18ec1f24f4f9691a5e7de5ff564" is_pre_election: true
I20260929 19:01:22.199328 8205 raft_consensus.cc:2454] T 9aec224873654d2ca51ece90012f8ecb P b27ba18ec1f24f4f9691a5e7de5ff564 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 649983f98f554789af84afcb77b8e38a in term 0.
I20260929 19:01:22.199380 8289 leader_election.cc:302] T 9aec224873654d2ca51ece90012f8ecb P 649983f98f554789af84afcb77b8e38a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 649983f98f554789af84afcb77b8e38a, 936984d765f745738bea4acbbe9c61fb; no voters:
I20260929 19:01:22.199496 8506 raft_consensus.cc:2781] T 9aec224873654d2ca51ece90012f8ecb P 649983f98f554789af84afcb77b8e38a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 19:01:22.199532 8506 raft_consensus.cc:482] T 9aec224873654d2ca51ece90012f8ecb P 649983f98f554789af84afcb77b8e38a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 19:01:22.199551 8506 raft_consensus.cc:3037] T 9aec224873654d2ca51ece90012f8ecb P 649983f98f554789af84afcb77b8e38a [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:01:22.200284 8506 raft_consensus.cc:504] T 9aec224873654d2ca51ece90012f8ecb P 649983f98f554789af84afcb77b8e38a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "936984d765f745738bea4acbbe9c61fb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37753 } } peers { permanent_uuid: "b27ba18ec1f24f4f9691a5e7de5ff564" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37627 } } peers { permanent_uuid: "649983f98f554789af84afcb77b8e38a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46835 } }
I20260929 19:01:22.200417 8506 leader_election.cc:288] T 9aec224873654d2ca51ece90012f8ecb P 649983f98f554789af84afcb77b8e38a [CANDIDATE]: Term 1 election: Requested vote from peers 936984d765f745738bea4acbbe9c61fb (127.0.98.1:37753), b27ba18ec1f24f4f9691a5e7de5ff564 (127.0.98.2:37627)
I20260929 19:01:22.200577 8205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9aec224873654d2ca51ece90012f8ecb" candidate_uuid: "649983f98f554789af84afcb77b8e38a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b27ba18ec1f24f4f9691a5e7de5ff564"
I20260929 19:01:22.200579 8076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9aec224873654d2ca51ece90012f8ecb" candidate_uuid: "649983f98f554789af84afcb77b8e38a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "936984d765f745738bea4acbbe9c61fb"
I20260929 19:01:22.200649 8205 raft_consensus.cc:3037] T 9aec224873654d2ca51ece90012f8ecb P b27ba18ec1f24f4f9691a5e7de5ff564 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:01:22.200649 8076 raft_consensus.cc:3037] T 9aec224873654d2ca51ece90012f8ecb P 936984d765f745738bea4acbbe9c61fb [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:01:22.201406 8076 raft_consensus.cc:2454] T 9aec224873654d2ca51ece90012f8ecb P 936984d765f745738bea4acbbe9c61fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 649983f98f554789af84afcb77b8e38a in term 1.
I20260929 19:01:22.201406 8205 raft_consensus.cc:2454] T 9aec224873654d2ca51ece90012f8ecb P b27ba18ec1f24f4f9691a5e7de5ff564 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 649983f98f554789af84afcb77b8e38a in term 1.
I20260929 19:01:22.201612 8290 leader_election.cc:302] T 9aec224873654d2ca51ece90012f8ecb P 649983f98f554789af84afcb77b8e38a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 649983f98f554789af84afcb77b8e38a, b27ba18ec1f24f4f9691a5e7de5ff564; no voters:
I20260929 19:01:22.201747 8506 raft_consensus.cc:2781] T 9aec224873654d2ca51ece90012f8ecb P 649983f98f554789af84afcb77b8e38a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:01:22.201805 8506 raft_consensus.cc:686] T 9aec224873654d2ca51ece90012f8ecb P 649983f98f554789af84afcb77b8e38a [term 1 LEADER]: Becoming Leader. State: Replica: 649983f98f554789af84afcb77b8e38a, State: Running, Role: LEADER
I20260929 19:01:22.201880 8506 consensus_queue.cc:237] T 9aec224873654d2ca51ece90012f8ecb P 649983f98f554789af84afcb77b8e38a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "936984d765f745738bea4acbbe9c61fb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37753 } } peers { permanent_uuid: "b27ba18ec1f24f4f9691a5e7de5ff564" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37627 } } peers { permanent_uuid: "649983f98f554789af84afcb77b8e38a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46835 } }
I20260929 19:01:22.202483 7918 catalog_manager.cc:5495] T 9aec224873654d2ca51ece90012f8ecb P 649983f98f554789af84afcb77b8e38a reported cstate change: term changed from 0 to 1, leader changed from <none> to 649983f98f554789af84afcb77b8e38a (127.0.98.3). New cstate: current_term: 1 leader_uuid: "649983f98f554789af84afcb77b8e38a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "936984d765f745738bea4acbbe9c61fb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37753 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b27ba18ec1f24f4f9691a5e7de5ff564" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37627 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "649983f98f554789af84afcb77b8e38a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46835 } health_report { overall_health: HEALTHY } } }
I20260929 19:01:22.216153 8428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:22.213932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51915 (local address 127.0.98.1:37753)
0929 19:01:22.213976 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:22.213979 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:22.214104 (+ 125us) server_negotiation.cc:407] Connection header received
0929 19:01:22.214172 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:22.214173 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:22.214181 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:22.214224 (+ 43us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 19:01:22.214316 (+ 92us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:22.214815 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:22.215579 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:22.215751 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:22.215803 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:22.215968 (+ 165us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 19:01:22.215992 (+ 24us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 19:01:22.216029 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0929 19:01:22.216080 (+ 51us) server_negotiation.cc:299] Negotiation successful
0929 19:01:22.216097 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260929 19:01:22.216158 8518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:22.213820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37753 (local address 127.0.98.3:51915)
0929 19:01:22.214039 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:22.214049 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 19:01:22.214104 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:01:22.214227 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:01:22.214227 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:01:22.214229 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 19:01:22.214269 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:22.214271 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:22.214877 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:22.214878 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:22.215508 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:22.215510 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:22.215812 (+ 302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:22.215813 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:22.215866 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:22.215877 (+ 11us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 19:01:22.216040 (+ 163us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 19:01:22.216041 (+ 1us) client_negotiation.cc:769] Sending connection context
0929 19:01:22.216071 (+ 30us) client_negotiation.cc:240] Negotiation successful
0929 19:01:22.216104 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":111,"threads_started":1}
I20260929 19:01:22.218019 8205 raft_consensus.cc:1261] T 9aec224873654d2ca51ece90012f8ecb P b27ba18ec1f24f4f9691a5e7de5ff564 [term 1 FOLLOWER]: Refusing update from remote peer 649983f98f554789af84afcb77b8e38a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 19:01:22.218024 8076 raft_consensus.cc:1261] T 9aec224873654d2ca51ece90012f8ecb P 936984d765f745738bea4acbbe9c61fb [term 1 FOLLOWER]: Refusing update from remote peer 649983f98f554789af84afcb77b8e38a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 19:01:22.218242 8511 consensus_queue.cc:1035] T 9aec224873654d2ca51ece90012f8ecb P 649983f98f554789af84afcb77b8e38a [LEADER]: Connected to new peer: Peer: permanent_uuid: "936984d765f745738bea4acbbe9c61fb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 19:01:22.218861 8511 consensus_queue.cc:1035] T 9aec224873654d2ca51ece90012f8ecb P 649983f98f554789af84afcb77b8e38a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b27ba18ec1f24f4f9691a5e7de5ff564" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37627 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 19:01:22.225116 8427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:22.222957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60385 (local address 127.0.98.3:46835)
0929 19:01:22.223029 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:22.223031 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:22.223049 (+ 18us) server_negotiation.cc:407] Connection header received
0929 19:01:22.223069 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:22.223070 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:22.223079 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:22.223126 (+ 47us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 19:01:22.223223 (+ 97us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:22.223655 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:22.224348 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:22.224599 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:22.224681 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:22.224853 (+ 172us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 19:01:22.224892 (+ 39us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 19:01:22.224971 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0929 19:01:22.225035 (+ 64us) server_negotiation.cc:299] Negotiation successful
0929 19:01:22.225053 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260929 19:01:22.225138 8488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:22.222814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46835 (local address 127.0.98.1:60385)
0929 19:01:22.222849 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:22.222860 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 19:01:22.222895 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:01:22.223129 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:01:22.223130 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:01:22.223131 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 19:01:22.223168 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:22.223169 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:22.223755 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:22.223756 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:22.224263 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:22.224264 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:22.224682 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:22.224682 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:22.224730 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:22.224736 (+ 6us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 19:01:22.224976 (+ 240us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 19:01:22.224978 (+ 2us) client_negotiation.cc:769] Sending connection context
0929 19:01:22.225030 (+ 52us) client_negotiation.cc:240] Negotiation successful
0929 19:01:22.225070 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14}
I20260929 19:01:22.231246 7918 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38788:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 19:01:22.231323 7918 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38788:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 19:01:22.231870 7918 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1db5b3fcae274bbf94cb5d9e6eff70fa: Sending DeleteTablet for 3 replicas of tablet 9aec224873654d2ca51ece90012f8ecb
I20260929 19:01:22.232121 8196 tablet_service.cc:1514] Processing DeleteTablet for tablet 9aec224873654d2ca51ece90012f8ecb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:01:22 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60092
I20260929 19:01:22.232121 8056 tablet_service.cc:1514] Processing DeleteTablet for tablet 9aec224873654d2ca51ece90012f8ecb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:01:22 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54914
I20260929 19:01:22.232178 8336 tablet_service.cc:1514] Processing DeleteTablet for tablet 9aec224873654d2ca51ece90012f8ecb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:01:22 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42396
I20260929 19:01:22.232218 8455 tablet_replica.cc:330] T 9aec224873654d2ca51ece90012f8ecb P 936984d765f745738bea4acbbe9c61fb: stopping tablet replica
I20260929 19:01:22.232262 8455 raft_consensus.cc:2229] T 9aec224873654d2ca51ece90012f8ecb P 936984d765f745738bea4acbbe9c61fb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 19:01:22.232271 8453 tablet_replica.cc:330] T 9aec224873654d2ca51ece90012f8ecb P 649983f98f554789af84afcb77b8e38a: stopping tablet replica
I20260929 19:01:22.232292 8455 raft_consensus.cc:2258] T 9aec224873654d2ca51ece90012f8ecb P 936984d765f745738bea4acbbe9c61fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 19:01:22.232324 8453 raft_consensus.cc:2229] T 9aec224873654d2ca51ece90012f8ecb P 649983f98f554789af84afcb77b8e38a [term 1 LEADER]: Raft consensus shutting down.
I20260929 19:01:22.232352 8454 tablet_replica.cc:330] T 9aec224873654d2ca51ece90012f8ecb P b27ba18ec1f24f4f9691a5e7de5ff564: stopping tablet replica
I20260929 19:01:22.232390 8453 raft_consensus.cc:2258] T 9aec224873654d2ca51ece90012f8ecb P 649983f98f554789af84afcb77b8e38a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 19:01:22.232424 8454 raft_consensus.cc:2229] T 9aec224873654d2ca51ece90012f8ecb P b27ba18ec1f24f4f9691a5e7de5ff564 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 19:01:22.232467 8454 raft_consensus.cc:2258] T 9aec224873654d2ca51ece90012f8ecb P b27ba18ec1f24f4f9691a5e7de5ff564 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 19:01:22.232826 8454 ts_tablet_manager.cc:1905] T 9aec224873654d2ca51ece90012f8ecb P b27ba18ec1f24f4f9691a5e7de5ff564: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 19:01:22.232921 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUbnLHE/build/release/bin/kudu with pid 7991
I20260929 19:01:22.234542 8454 ts_tablet_manager.cc:1918] T 9aec224873654d2ca51ece90012f8ecb P b27ba18ec1f24f4f9691a5e7de5ff564: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 19:01:22.234596 8454 log.cc:1196] T 9aec224873654d2ca51ece90012f8ecb P b27ba18ec1f24f4f9691a5e7de5ff564: Deleting WAL directory at /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708447806542-392-0/minicluster-data/ts-1/wal/wals/9aec224873654d2ca51ece90012f8ecb
I20260929 19:01:22.234843 8454 ts_tablet_manager.cc:1939] T 9aec224873654d2ca51ece90012f8ecb P b27ba18ec1f24f4f9691a5e7de5ff564: Deleting consensus metadata
W20260929 19:01:22.239928 7905 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:37753 (error 108)
I20260929 19:01:22.240023 7906 catalog_manager.cc:4841] TS b27ba18ec1f24f4f9691a5e7de5ff564 (127.0.98.2:37627): tablet 9aec224873654d2ca51ece90012f8ecb (table test-table [id=0388bd084fb44761b49564b16e15bc35]) successfully deleted
I20260929 19:01:22.240181 8453 ts_tablet_manager.cc:1905] T 9aec224873654d2ca51ece90012f8ecb P 649983f98f554789af84afcb77b8e38a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 19:01:22.240437 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUbnLHE/build/release/bin/kudu with pid 8131
I20260929 19:01:22.240507 8522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:22.240304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37753 (local address 127.0.0.1:54924)
0929 19:01:22.240389 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:22.240432 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72,"thread_start_us":45,"threads_started":1}
I20260929 19:01:22.241963 8453 ts_tablet_manager.cc:1918] T 9aec224873654d2ca51ece90012f8ecb P 649983f98f554789af84afcb77b8e38a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 19:01:22.242028 8453 log.cc:1196] T 9aec224873654d2ca51ece90012f8ecb P 649983f98f554789af84afcb77b8e38a: Deleting WAL directory at /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790708447806542-392-0/minicluster-data/ts-2/wal/wals/9aec224873654d2ca51ece90012f8ecb
I20260929 19:01:22.242230 8453 ts_tablet_manager.cc:1939] T 9aec224873654d2ca51ece90012f8ecb P 649983f98f554789af84afcb77b8e38a: Deleting consensus metadata
I20260929 19:01:22.242602 7903 catalog_manager.cc:4841] TS 649983f98f554789af84afcb77b8e38a (127.0.98.3:46835): tablet 9aec224873654d2ca51ece90012f8ecb (table test-table [id=0388bd084fb44761b49564b16e15bc35]) successfully deleted
W20260929 19:01:22.243130 7905 catalog_manager.cc:4584] TS 936984d765f745738bea4acbbe9c61fb (127.0.98.1:37753): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9aec224873654d2ca51ece90012f8ecb: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37753: connect: Connection refused (error 111)
I20260929 19:01:22.247310 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUbnLHE/build/release/bin/kudu with pid 8271
I20260929 19:01:22.252861 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUbnLHE/build/release/bin/kudu with pid 7887
2026-09-29T19:01:22Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (2726 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790708447806542-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 19:01:22 dist-test-slave-lqkv krb5kdc[8525](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 19:01:22 dist-test-slave-lqkv krb5kdc[8525](info): set up 2 sockets
Sep 29 19:01:22 dist-test-slave-lqkv krb5kdc[8525](info): commencing operation
krb5kdc: starting...
W20260929 19:01:23.289244 392 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790708447806542-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790708447806542-392-0/krb5kdc/test-user.keytab.
Sep 29 19:01:23 dist-test-slave-lqkv krb5kdc[8525](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708483, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T19:01:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T19:01:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790708447806542-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790708447806542-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 19:01:23.399881 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUbnLHE/build/release/bin/kudu
/tmp/dist-test-taskUbnLHE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790708447806542-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790708447806542-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790708447806542-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790708447806542-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33221
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790708447806542-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33221
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790708447806542-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790708447806542-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790708447806542-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:01:23.476356 8538 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:01:23.476568 8538 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:01:23.476604 8538 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:01:23.477720 8538 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 19:01:23.477784 8538 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:01:23.477808 8538 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:01:23.477836 8538 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 19:01:23.477866 8538 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 19:01:23.479315 8538 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790708447806542-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790708447806542-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33221
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790708447806542-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33221
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790708447806542-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8538
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790708447806542-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 18:09:13 UTC on ccycloud.quasar-hpkalf.root.comops.site
build id 4680913
I20260929 19:01:23.479550 8538 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:01:23.479801 8538 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:01:23.484488 8544 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:01:23.484476 8543 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:01:23.484719 8538 server_base.cc:1034] running on GCE node
W20260929 19:01:23.484476 8546 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:01:23.485021 8538 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:01:23.485327 8538 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:01:23.486522 8538 hybrid_clock.cc:648] HybridClock initialized: now 1790708483486507 us; error 25 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-29T19:01:23Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1243 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708447806542-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 19:01:23 dist-test-slave-lqkv krb5kdc[8551](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 19:01:23 dist-test-slave-lqkv krb5kdc[8551](info): set up 2 sockets
Sep 29 19:01:23 dist-test-slave-lqkv krb5kdc[8551](info): commencing operation
krb5kdc: starting...
W20260929 19:01:24.533301 392 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708447806542-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708447806542-392-0/krb5kdc/test-user.keytab.
Sep 29 19:01:24 dist-test-slave-lqkv krb5kdc[8551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708484, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T19:01:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T19:01:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708447806542-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708447806542-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 19:01:24.638621 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUbnLHE/build/release/bin/kudu
/tmp/dist-test-taskUbnLHE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708447806542-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708447806542-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708447806542-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708447806542-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35045
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708447806542-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35045
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708447806542-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708447806542-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708447806542-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:01:24.714437 8564 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:01:24.714638 8564 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:01:24.714684 8564 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:01:24.715756 8564 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 19:01:24.715814 8564 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:01:24.715839 8564 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:01:24.715868 8564 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 19:01:24.715890 8564 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 19:01:24.717342 8564 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708447806542-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708447806542-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35045
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708447806542-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35045
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708447806542-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8564
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708447806542-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 18:09:13 UTC on ccycloud.quasar-hpkalf.root.comops.site
build id 4680913
I20260929 19:01:24.717590 8564 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:01:24.717790 8564 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:01:24.722179 8572 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:01:24.722198 8570 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:01:24.722188 8569 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:01:24.722357 8564 server_base.cc:1034] running on GCE node
I20260929 19:01:24.722584 8564 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:01:24.722900 8564 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:01:24.724059 8564 hybrid_clock.cc:648] HybridClock initialized: now 1790708484724045 us; error 40 us; skew 500 ppm
Sep 29 19:01:24 dist-test-slave-lqkv krb5kdc[8551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708484, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:01:24.730135 8564 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 19:01:24.730530 8564 webserver.cc:460] Webserver started at http://127.0.98.62:41897/ using document root <none> and password file <none>
I20260929 19:01:24.730793 8564 fs_manager.cc:359] Metadata directory not provided
I20260929 19:01:24.730858 8564 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:01:24.730999 8564 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:01:24.731967 8564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708447806542-392-0/minicluster-data/master-0/data/instance:
uuid: "d19a32a47cb04612acc8e39a0386dc70"
format_stamp: "Formatted at 2026-09-29 19:01:24 on dist-test-slave-lqkv"
server_key: "228018d460feea827ced5c4bad5f303b"
server_key_iv: "73e18ae369ac3de8e9d5d6a55bf80db8"
server_key_version: "encryptionkey@0"
I20260929 19:01:24.732326 8564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708447806542-392-0/minicluster-data/master-0/wal/instance:
uuid: "d19a32a47cb04612acc8e39a0386dc70"
format_stamp: "Formatted at 2026-09-29 19:01:24 on dist-test-slave-lqkv"
server_key: "228018d460feea827ced5c4bad5f303b"
server_key_iv: "73e18ae369ac3de8e9d5d6a55bf80db8"
server_key_version: "encryptionkey@0"
I20260929 19:01:24.733673 8564 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 19:01:24.734346 8579 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:24.734514 8564 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 19:01:24.734608 8564 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708447806542-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708447806542-392-0/minicluster-data/master-0/wal
uuid: "d19a32a47cb04612acc8e39a0386dc70"
format_stamp: "Formatted at 2026-09-29 19:01:24 on dist-test-slave-lqkv"
server_key: "228018d460feea827ced5c4bad5f303b"
server_key_iv: "73e18ae369ac3de8e9d5d6a55bf80db8"
server_key_version: "encryptionkey@0"
I20260929 19:01:24.734685 8564 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708447806542-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708447806542-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708447806542-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:01:24.756127 8564 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:01:24.757831 8564 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:01:24.757943 8564 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:01:24.761390 8564 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35045
I20260929 19:01:24.761437 8631 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35045 every 8 connection(s)
I20260929 19:01:24.761873 8564 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708447806542-392-0/minicluster-data/master-0/data/info.pb
I20260929 19:01:24.762478 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUbnLHE/build/release/bin/kudu as pid 8564
I20260929 19:01:24.762512 8632 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:01:24.762590 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708447806542-392-0/minicluster-data/master-0/wal/instance
I20260929 19:01:24.762795 392 external_mini_cluster.cc:1428] Setting key 08aa32fe4ad4c0a856c7766187751a11
I20260929 19:01:24.765456 8632 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d19a32a47cb04612acc8e39a0386dc70: Bootstrap starting.
I20260929 19:01:24.766121 8632 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d19a32a47cb04612acc8e39a0386dc70: Neither blocks nor log segments found. Creating new log.
I20260929 19:01:24.766420 8632 log.cc:824] T 00000000000000000000000000000000 P d19a32a47cb04612acc8e39a0386dc70: Log is configured to *not* fsync() on all Append() calls
I20260929 19:01:24.767069 8632 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d19a32a47cb04612acc8e39a0386dc70: No bootstrap required, opened a new log
Sep 29 19:01:24 dist-test-slave-lqkv krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708484, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 19:01:24.769659 8632 raft_consensus.cc:348] T 00000000000000000000000000000000 P d19a32a47cb04612acc8e39a0386dc70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d19a32a47cb04612acc8e39a0386dc70" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35045 } }
I20260929 19:01:24.769837 8632 raft_consensus.cc:374] T 00000000000000000000000000000000 P d19a32a47cb04612acc8e39a0386dc70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:01:24.769876 8632 raft_consensus.cc:729] T 00000000000000000000000000000000 P d19a32a47cb04612acc8e39a0386dc70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d19a32a47cb04612acc8e39a0386dc70, State: Initialized, Role: FOLLOWER
I20260929 19:01:24.770002 8632 consensus_queue.cc:260] T 00000000000000000000000000000000 P d19a32a47cb04612acc8e39a0386dc70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d19a32a47cb04612acc8e39a0386dc70" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35045 } }
I20260929 19:01:24.770054 8632 raft_consensus.cc:388] T 00000000000000000000000000000000 P d19a32a47cb04612acc8e39a0386dc70 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 19:01:24.770095 8632 raft_consensus.cc:482] T 00000000000000000000000000000000 P d19a32a47cb04612acc8e39a0386dc70 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 19:01:24.770139 8632 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d19a32a47cb04612acc8e39a0386dc70 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:01:24.770788 8632 raft_consensus.cc:504] T 00000000000000000000000000000000 P d19a32a47cb04612acc8e39a0386dc70 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d19a32a47cb04612acc8e39a0386dc70" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35045 } }
I20260929 19:01:24.770885 8632 leader_election.cc:302] T 00000000000000000000000000000000 P d19a32a47cb04612acc8e39a0386dc70 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d19a32a47cb04612acc8e39a0386dc70; no voters:
I20260929 19:01:24.771018 8632 leader_election.cc:288] T 00000000000000000000000000000000 P d19a32a47cb04612acc8e39a0386dc70 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 19:01:24.771098 8637 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d19a32a47cb04612acc8e39a0386dc70 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:01:24.771255 8637 raft_consensus.cc:686] T 00000000000000000000000000000000 P d19a32a47cb04612acc8e39a0386dc70 [term 1 LEADER]: Becoming Leader. State: Replica: d19a32a47cb04612acc8e39a0386dc70, State: Running, Role: LEADER
I20260929 19:01:24.771265 8632 sys_catalog.cc:564] T 00000000000000000000000000000000 P d19a32a47cb04612acc8e39a0386dc70 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 19:01:24.771409 8637 consensus_queue.cc:237] T 00000000000000000000000000000000 P d19a32a47cb04612acc8e39a0386dc70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d19a32a47cb04612acc8e39a0386dc70" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35045 } }
I20260929 19:01:24.771984 8638 sys_catalog.cc:455] T 00000000000000000000000000000000 P d19a32a47cb04612acc8e39a0386dc70 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d19a32a47cb04612acc8e39a0386dc70. Latest consensus state: current_term: 1 leader_uuid: "d19a32a47cb04612acc8e39a0386dc70" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d19a32a47cb04612acc8e39a0386dc70" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35045 } } }
I20260929 19:01:24.772058 8638 sys_catalog.cc:458] T 00000000000000000000000000000000 P d19a32a47cb04612acc8e39a0386dc70 [sys.catalog]: This master's current role is: LEADER
I20260929 19:01:24.771974 8639 sys_catalog.cc:455] T 00000000000000000000000000000000 P d19a32a47cb04612acc8e39a0386dc70 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d19a32a47cb04612acc8e39a0386dc70" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d19a32a47cb04612acc8e39a0386dc70" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35045 } } }
I20260929 19:01:24.772373 8639 sys_catalog.cc:458] T 00000000000000000000000000000000 P d19a32a47cb04612acc8e39a0386dc70 [sys.catalog]: This master's current role is: LEADER
I20260929 19:01:24.772437 8646 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 19:01:24.772960 8646 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 19:01:24.773530 8635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:24.763251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57526 (local address 127.0.98.62:35045)
0929 19:01:24.763420 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:24.763427 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:24.763449 (+ 22us) server_negotiation.cc:407] Connection header received
0929 19:01:24.763894 (+ 445us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:24.763900 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:24.763940 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:24.764040 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:01:24.764784 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:24.765708 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:24.766759 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:24.767183 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:24.767288 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:24.769707 (+ 2419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:01:24.769710 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:01:24.769711 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:01:24.769721 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:01:24.771507 (+ 1786us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:24.771894 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:24.771897 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:24.771898 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:24.772010 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:24.772202 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:24.772203 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:24.772203 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:24.772487 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:01:24.772685 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0929 19:01:24.773225 (+ 540us) server_negotiation.cc:299] Negotiation successful
0929 19:01:24.773349 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":91,"threads_started":1}
I20260929 19:01:24.775117 8646 catalog_manager.cc:1347] Generated new cluster ID: 930b2aeca3bf4076ba1fb8c667371fb0
I20260929 19:01:24.775163 8646 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 19:01:24.782518 8646 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 19:01:24.783075 8646 catalog_manager.cc:1504] Loading token signing keys...
I20260929 19:01:24.788393 8646 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d19a32a47cb04612acc8e39a0386dc70: Generated new TSK 0
I20260929 19:01:24.788554 8646 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708447806542-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708447806542-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260929 19:01:24.814991 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUbnLHE/build/release/bin/kudu
/tmp/dist-test-taskUbnLHE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708447806542-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708447806542-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708447806542-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708447806542-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35045
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708447806542-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708447806542-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708447806542-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708447806542-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:01:24.927870 8658 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:01:24.928109 8658 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:01:24.928140 8658 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:01:24.929772 8658 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:01:24.929829 8658 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:01:24.929865 8658 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 19:01:24.931941 8658 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708447806542-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708447806542-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708447806542-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708447806542-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35045
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8658
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708447806542-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 18:09:13 UTC on ccycloud.quasar-hpkalf.root.comops.site
build id 4680913
I20260929 19:01:24.932159 8658 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:01:24.932423 8658 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:01:24.937281 8666 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:01:24.937281 8664 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:01:24.937333 8663 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:01:24.937493 8658 server_base.cc:1034] running on GCE node
I20260929 19:01:24.937688 8658 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:01:24.937963 8658 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:01:24.939123 8658 hybrid_clock.cc:648] HybridClock initialized: now 1790708484939107 us; error 36 us; skew 500 ppm
Sep 29 19:01:24 dist-test-slave-lqkv krb5kdc[8551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708484, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:01:24.941520 8658 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 19:01:24.942037 8658 webserver.cc:460] Webserver started at http://127.0.98.1:44093/ using document root <none> and password file <none>
I20260929 19:01:24.942262 8658 fs_manager.cc:359] Metadata directory not provided
I20260929 19:01:24.942327 8658 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:01:24.942482 8658 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:01:24.943435 8658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708447806542-392-0/minicluster-data/ts-0/data/instance:
uuid: "b05badcafa5a4848a22be4b00c2f0bab"
format_stamp: "Formatted at 2026-09-29 19:01:24 on dist-test-slave-lqkv"
server_key: "e5be03752a4b90413e868cf856055f18"
server_key_iv: "031aac9a36978953127d1f39c0ea8075"
server_key_version: "encryptionkey@0"
I20260929 19:01:24.943769 8658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708447806542-392-0/minicluster-data/ts-0/wal/instance:
uuid: "b05badcafa5a4848a22be4b00c2f0bab"
format_stamp: "Formatted at 2026-09-29 19:01:24 on dist-test-slave-lqkv"
server_key: "e5be03752a4b90413e868cf856055f18"
server_key_iv: "031aac9a36978953127d1f39c0ea8075"
server_key_version: "encryptionkey@0"
I20260929 19:01:24.945066 8658 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260929 19:01:24.945735 8673 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:24.945930 8658 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:24.946019 8658 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708447806542-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708447806542-392-0/minicluster-data/ts-0/wal
uuid: "b05badcafa5a4848a22be4b00c2f0bab"
format_stamp: "Formatted at 2026-09-29 19:01:24 on dist-test-slave-lqkv"
server_key: "e5be03752a4b90413e868cf856055f18"
server_key_iv: "031aac9a36978953127d1f39c0ea8075"
server_key_version: "encryptionkey@0"
I20260929 19:01:24.946094 8658 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708447806542-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708447806542-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708447806542-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:01:24.954576 8658 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:01:24.956147 8658 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:01:24.956295 8658 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:01:24.956488 8658 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:01:24.956766 8658 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:01:24.956866 8658 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:24.956925 8658 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:01:24.956956 8658 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:24.962575 8658 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33311
I20260929 19:01:24.962617 8786 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33311 every 8 connection(s)
I20260929 19:01:24.962942 8658 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708447806542-392-0/minicluster-data/ts-0/data/info.pb
Sep 29 19:01:24 dist-test-slave-lqkv krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708484, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 19:01:24.971652 8635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:24.963601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38695 (local address 127.0.98.62:35045)
0929 19:01:24.963639 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:24.963641 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:24.963860 (+ 219us) server_negotiation.cc:407] Connection header received
0929 19:01:24.964730 (+ 870us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:24.964731 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:24.964744 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:24.964798 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:01:24.965412 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:24.965906 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:24.966925 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:24.967134 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:24.967218 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:24.969202 (+ 1984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:01:24.969203 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:01:24.969204 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:01:24.969211 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:01:24.970443 (+ 1232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:24.970819 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:24.970820 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:24.970821 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:24.970869 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:24.971074 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:24.971074 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:24.971075 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:24.971165 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:01:24.971212 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0929 19:01:24.971528 (+ 316us) server_negotiation.cc:299] Negotiation successful
0929 19:01:24.971590 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260929 19:01:24.971851 8789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:24.963580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35045 (local address 127.0.98.1:38695)
0929 19:01:24.963749 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:24.963778 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0929 19:01:24.964628 (+ 850us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:01:24.964885 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:01:24.964888 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:01:24.965147 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:01:24.965308 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:24.965311 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:24.965996 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:24.965997 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:24.966807 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:24.966809 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:24.967226 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:24.967227 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:24.967353 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:24.967761 (+ 408us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:01:24.967768 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:01:24.969122 (+ 1354us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:01:24.970515 (+ 1393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:01:24.970520 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:01:24.970522 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:01:24.970737 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:01:24.970926 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:01:24.970927 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:01:24.970927 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:01:24.971020 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:01:24.971218 (+ 198us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:01:24.971222 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:01:24.971420 (+ 198us) client_negotiation.cc:769] Sending connection context
0929 19:01:24.971503 (+ 83us) client_negotiation.cc:240] Negotiation successful
0929 19:01:24.971651 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":90,"threads_started":1}
I20260929 19:01:24.972271 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUbnLHE/build/release/bin/kudu as pid 8658
I20260929 19:01:24.972329 8787 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35045
I20260929 19:01:24.972407 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708447806542-392-0/minicluster-data/ts-0/wal/instance
I20260929 19:01:24.972450 8787 heartbeater.cc:464] Registering TS with master...
I20260929 19:01:24.972612 392 external_mini_cluster.cc:1428] Setting key cf94295f0061ba6b14aca6d27c2f7532
I20260929 19:01:24.972610 8787 heartbeater.cc:511] Master 127.0.98.62:35045 requested a full tablet report, sending...
I20260929 19:01:24.973170 8596 ts_manager.cc:194] Registered new tserver with Master: b05badcafa5a4848a22be4b00c2f0bab (127.0.98.1:33311)
I20260929 19:01:24.974053 8596 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:38695
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708447806542-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708447806542-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260929 19:01:24.995114 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUbnLHE/build/release/bin/kudu
/tmp/dist-test-taskUbnLHE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708447806542-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708447806542-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708447806542-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708447806542-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35045
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708447806542-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708447806542-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708447806542-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708447806542-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:01:25.071477 8792 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:01:25.071707 8792 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:01:25.071743 8792 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:01:25.073060 8792 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:01:25.073117 8792 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:01:25.073163 8792 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 19:01:25.074579 8792 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708447806542-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708447806542-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708447806542-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708447806542-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35045
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8792
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708447806542-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 18:09:13 UTC on ccycloud.quasar-hpkalf.root.comops.site
build id 4680913
I20260929 19:01:25.074829 8792 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:01:25.075065 8792 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:01:25.079694 8798 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:01:25.079702 8797 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:01:25.079708 8800 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:01:25.079965 8792 server_base.cc:1034] running on GCE node
I20260929 19:01:25.080214 8792 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:01:25.080451 8792 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:01:25.081601 8792 hybrid_clock.cc:648] HybridClock initialized: now 1790708485081580 us; error 37 us; skew 500 ppm
Sep 29 19:01:25 dist-test-slave-lqkv krb5kdc[8551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708485, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:01:25.083865 8792 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 19:01:25.084301 8792 webserver.cc:460] Webserver started at http://127.0.98.2:32911/ using document root <none> and password file <none>
I20260929 19:01:25.084507 8792 fs_manager.cc:359] Metadata directory not provided
I20260929 19:01:25.084571 8792 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:01:25.084712 8792 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:01:25.085706 8792 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708447806542-392-0/minicluster-data/ts-1/data/instance:
uuid: "d02737d125f74c97adb08635f60a7ade"
format_stamp: "Formatted at 2026-09-29 19:01:25 on dist-test-slave-lqkv"
server_key: "d20fea4f680f1e7d2f0fffb7188121c1"
server_key_iv: "21d6cff243a65371c86e0b1b319afea0"
server_key_version: "encryptionkey@0"
I20260929 19:01:25.086054 8792 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708447806542-392-0/minicluster-data/ts-1/wal/instance:
uuid: "d02737d125f74c97adb08635f60a7ade"
format_stamp: "Formatted at 2026-09-29 19:01:25 on dist-test-slave-lqkv"
server_key: "d20fea4f680f1e7d2f0fffb7188121c1"
server_key_iv: "21d6cff243a65371c86e0b1b319afea0"
server_key_version: "encryptionkey@0"
I20260929 19:01:25.087260 8792 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 19:01:25.087885 8807 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:25.088040 8792 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 19:01:25.088130 8792 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708447806542-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708447806542-392-0/minicluster-data/ts-1/wal
uuid: "d02737d125f74c97adb08635f60a7ade"
format_stamp: "Formatted at 2026-09-29 19:01:25 on dist-test-slave-lqkv"
server_key: "d20fea4f680f1e7d2f0fffb7188121c1"
server_key_iv: "21d6cff243a65371c86e0b1b319afea0"
server_key_version: "encryptionkey@0"
I20260929 19:01:25.088223 8792 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708447806542-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708447806542-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708447806542-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:01:25.097728 8792 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:01:25.099340 8792 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:01:25.099496 8792 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:01:25.099701 8792 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:01:25.099987 8792 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:01:25.100036 8792 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:25.100069 8792 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:01:25.100095 8792 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:25.105718 8792 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34695
I20260929 19:01:25.105770 8920 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34695 every 8 connection(s)
I20260929 19:01:25.106070 8792 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708447806542-392-0/minicluster-data/ts-1/data/info.pb
I20260929 19:01:25.109213 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUbnLHE/build/release/bin/kudu as pid 8792
I20260929 19:01:25.109318 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708447806542-392-0/minicluster-data/ts-1/wal/instance
I20260929 19:01:25.109520 392 external_mini_cluster.cc:1428] Setting key f825c065422534570525d59d32ab0beb
Sep 29 19:01:25 dist-test-slave-lqkv krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708485, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 19:01:25.115973 8635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:25.106793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39141 (local address 127.0.98.62:35045)
0929 19:01:25.106836 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:25.106838 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:25.107014 (+ 176us) server_negotiation.cc:407] Connection header received
0929 19:01:25.107831 (+ 817us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:25.107832 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:25.107840 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:25.107894 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:01:25.108393 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:25.108895 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:25.110081 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:25.110394 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:25.110476 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:25.112631 (+ 2155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:01:25.112632 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:01:25.112632 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:01:25.112638 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:01:25.114524 (+ 1886us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:25.115042 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:25.115043 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:25.115044 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:25.115092 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:25.115363 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:25.115363 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:25.115364 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:25.115460 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:01:25.115513 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0929 19:01:25.115824 (+ 311us) server_negotiation.cc:299] Negotiation successful
0929 19:01:25.115892 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260929 19:01:25.116236 8923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:25.106739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35045 (local address 127.0.98.2:39141)
0929 19:01:25.106883 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:25.106928 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 19:01:25.107750 (+ 822us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:01:25.107945 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:01:25.107948 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:01:25.108167 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:01:25.108325 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:25.108328 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:25.108965 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:25.108965 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:25.109974 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:25.109976 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:25.110511 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:25.110513 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:25.110702 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:25.111265 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:01:25.111275 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:01:25.112543 (+ 1268us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:01:25.114635 (+ 2092us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:01:25.114641 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:01:25.114643 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:01:25.114961 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:01:25.115155 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:01:25.115156 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:01:25.115156 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:01:25.115295 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:01:25.115524 (+ 229us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:01:25.115530 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:01:25.115692 (+ 162us) client_negotiation.cc:769] Sending connection context
0929 19:01:25.115812 (+ 120us) client_negotiation.cc:240] Negotiation successful
0929 19:01:25.115994 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":86,"threads_started":1}
I20260929 19:01:25.116757 8921 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35045
I20260929 19:01:25.116884 8921 heartbeater.cc:464] Registering TS with master...
I20260929 19:01:25.117061 8921 heartbeater.cc:511] Master 127.0.98.62:35045 requested a full tablet report, sending...
I20260929 19:01:25.117443 8596 ts_manager.cc:194] Registered new tserver with Master: d02737d125f74c97adb08635f60a7ade (127.0.98.2:34695)
I20260929 19:01:25.117820 8596 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:39141
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708447806542-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708447806542-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260929 19:01:25.133360 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUbnLHE/build/release/bin/kudu
/tmp/dist-test-taskUbnLHE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708447806542-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708447806542-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708447806542-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708447806542-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35045
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708447806542-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708447806542-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708447806542-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708447806542-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:01:25.210318 8926 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:01:25.210510 8926 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:01:25.210532 8926 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:01:25.211669 8926 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:01:25.211710 8926 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:01:25.211736 8926 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 19:01:25.213274 8926 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708447806542-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708447806542-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708447806542-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708447806542-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35045
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8926
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708447806542-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 18:09:13 UTC on ccycloud.quasar-hpkalf.root.comops.site
build id 4680913
I20260929 19:01:25.213482 8926 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:01:25.213718 8926 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:01:25.218293 8934 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:01:25.218281 8932 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:01:25.218282 8931 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:01:25.218582 8926 server_base.cc:1034] running on GCE node
I20260929 19:01:25.218801 8926 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:01:25.218984 8926 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:01:25.220144 8926 hybrid_clock.cc:648] HybridClock initialized: now 1790708485220134 us; error 42 us; skew 500 ppm
Sep 29 19:01:25 dist-test-slave-lqkv krb5kdc[8551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708485, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:01:25.222435 8926 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 19:01:25.222819 8926 webserver.cc:460] Webserver started at http://127.0.98.3:43945/ using document root <none> and password file <none>
I20260929 19:01:25.223062 8926 fs_manager.cc:359] Metadata directory not provided
I20260929 19:01:25.223127 8926 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:01:25.223274 8926 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:01:25.224207 8926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708447806542-392-0/minicluster-data/ts-2/data/instance:
uuid: "8c8f98470f884957951c584ac1c8ef10"
format_stamp: "Formatted at 2026-09-29 19:01:25 on dist-test-slave-lqkv"
server_key: "23d7367736ed14a64f500bd933d0c9ea"
server_key_iv: "a37499165678784128ce56242efd19ed"
server_key_version: "encryptionkey@0"
I20260929 19:01:25.224540 8926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708447806542-392-0/minicluster-data/ts-2/wal/instance:
uuid: "8c8f98470f884957951c584ac1c8ef10"
format_stamp: "Formatted at 2026-09-29 19:01:25 on dist-test-slave-lqkv"
server_key: "23d7367736ed14a64f500bd933d0c9ea"
server_key_iv: "a37499165678784128ce56242efd19ed"
server_key_version: "encryptionkey@0"
I20260929 19:01:25.225826 8926 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 19:01:25.226455 8941 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:25.226612 8926 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 19:01:25.226702 8926 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708447806542-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708447806542-392-0/minicluster-data/ts-2/wal
uuid: "8c8f98470f884957951c584ac1c8ef10"
format_stamp: "Formatted at 2026-09-29 19:01:25 on dist-test-slave-lqkv"
server_key: "23d7367736ed14a64f500bd933d0c9ea"
server_key_iv: "a37499165678784128ce56242efd19ed"
server_key_version: "encryptionkey@0"
I20260929 19:01:25.226785 8926 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708447806542-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708447806542-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708447806542-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:01:25.244081 8926 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:01:25.245790 8926 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:01:25.245937 8926 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:01:25.246151 8926 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:01:25.246419 8926 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:01:25.246471 8926 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:25.246505 8926 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:01:25.246532 8926 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:25.252027 8926 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44369
I20260929 19:01:25.252096 9054 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44369 every 8 connection(s)
I20260929 19:01:25.252372 8926 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708447806542-392-0/minicluster-data/ts-2/data/info.pb
I20260929 19:01:25.257342 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUbnLHE/build/release/bin/kudu as pid 8926
I20260929 19:01:25.257471 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708447806542-392-0/minicluster-data/ts-2/wal/instance
I20260929 19:01:25.257670 392 external_mini_cluster.cc:1428] Setting key 09fd1c5d1cc73e8c657a21f319fae3c0
Sep 29 19:01:25 dist-test-slave-lqkv krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708485, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 19:01:25.260694 8635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:25.253087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42171 (local address 127.0.98.62:35045)
0929 19:01:25.253159 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:25.253161 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:25.253320 (+ 159us) server_negotiation.cc:407] Connection header received
0929 19:01:25.254154 (+ 834us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:25.254155 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:25.254164 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:25.254213 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:01:25.254731 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:25.255189 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:25.256101 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:25.256286 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:25.256333 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:25.258509 (+ 2176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:01:25.258510 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:01:25.258511 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:01:25.258518 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:01:25.259654 (+ 1136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:25.259994 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:25.259995 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:25.259995 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:25.260042 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:25.260235 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:25.260236 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:25.260236 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:25.260325 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:01:25.260372 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0929 19:01:25.260583 (+ 211us) server_negotiation.cc:299] Negotiation successful
0929 19:01:25.260636 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260929 19:01:25.260885 9057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:25.253046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35045 (local address 127.0.98.3:42171)
0929 19:01:25.253206 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:25.253237 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0929 19:01:25.254055 (+ 818us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:01:25.254259 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:01:25.254262 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:01:25.254500 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:01:25.254667 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:25.254670 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:25.255257 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:25.255258 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:25.256022 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:25.256024 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:25.256351 (+ 327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:25.256352 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:25.256440 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:25.256892 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:01:25.256901 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:01:25.258393 (+ 1492us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:01:25.259718 (+ 1325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:01:25.259723 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:01:25.259724 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:01:25.259924 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:01:25.260097 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:01:25.260097 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:01:25.260097 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:01:25.260183 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:01:25.260375 (+ 192us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:01:25.260379 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:01:25.260473 (+ 94us) client_negotiation.cc:769] Sending connection context
0929 19:01:25.260554 (+ 81us) client_negotiation.cc:240] Negotiation successful
0929 19:01:25.260654 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":93,"threads_started":1}
I20260929 19:01:25.261356 9055 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35045
I20260929 19:01:25.261451 9055 heartbeater.cc:464] Registering TS with master...
I20260929 19:01:25.261620 9055 heartbeater.cc:511] Master 127.0.98.62:35045 requested a full tablet report, sending...
I20260929 19:01:25.261963 8596 ts_manager.cc:194] Registered new tserver with Master: 8c8f98470f884957951c584ac1c8ef10 (127.0.98.3:44369)
I20260929 19:01:25.262331 8596 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:42171
I20260929 19:01:25.269362 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708447806542-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708447806542-392-0/krb5kdc/oryx_127.0.98.1.keytab.
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708447806542-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708447806542-392-0/krb5kdc/oryx_127.0.98.2.keytab.
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708447806542-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708447806542-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260929 19:01:25.329114 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUbnLHE/build/release/bin/kudu with pid 8658
I20260929 19:01:25.333753 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUbnLHE/build/release/bin/kudu with pid 8792
I20260929 19:01:25.337802 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUbnLHE/build/release/bin/kudu with pid 8926
I20260929 19:01:25.341890 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUbnLHE/build/release/bin/kudu with pid 8564
I20260929 19:01:25.346459 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUbnLHE/build/release/bin/kudu
/tmp/dist-test-taskUbnLHE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708447806542-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708447806542-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708447806542-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708447806542-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35045
--webserver_interface=127.0.98.62
--webserver_port=41897
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708447806542-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35045
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708447806542-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708447806542-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708447806542-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:01:25.423583 9064 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:01:25.423766 9064 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:01:25.423789 9064 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:01:25.424856 9064 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 19:01:25.424888 9064 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:01:25.424902 9064 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:01:25.424914 9064 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 19:01:25.424926 9064 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 19:01:25.426270 9064 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708447806542-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708447806542-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35045
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708447806542-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35045
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708447806542-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=41897
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9064
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708447806542-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 18:09:13 UTC on ccycloud.quasar-hpkalf.root.comops.site
build id 4680913
I20260929 19:01:25.426425 9064 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:01:25.426573 9064 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:01:25.431066 9070 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:01:25.431026 9069 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:01:25.431034 9072 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:01:25.431262 9064 server_base.cc:1034] running on GCE node
I20260929 19:01:25.431519 9064 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:01:25.431849 9064 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:01:25.433027 9064 hybrid_clock.cc:648] HybridClock initialized: now 1790708485433009 us; error 41 us; skew 500 ppm
Sep 29 19:01:25 dist-test-slave-lqkv krb5kdc[8551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708485, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:01:25.435258 9064 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 19:01:25.435709 9064 webserver.cc:460] Webserver started at http://127.0.98.62:41897/ using document root <none> and password file <none>
I20260929 19:01:25.435938 9064 fs_manager.cc:359] Metadata directory not provided
I20260929 19:01:25.436017 9064 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:01:25.437335 9064 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 19:01:25.437948 9079 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:25.438120 9064 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 19:01:25.438212 9064 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708447806542-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708447806542-392-0/minicluster-data/master-0/wal
uuid: "d19a32a47cb04612acc8e39a0386dc70"
format_stamp: "Formatted at 2026-09-29 19:01:24 on dist-test-slave-lqkv"
server_key: "228018d460feea827ced5c4bad5f303b"
server_key_iv: "73e18ae369ac3de8e9d5d6a55bf80db8"
server_key_version: "encryptionkey@0"
I20260929 19:01:25.438510 9064 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708447806542-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708447806542-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708447806542-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:01:25.445801 9064 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:01:25.447355 9064 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:01:25.447496 9064 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:01:25.451282 9064 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35045
I20260929 19:01:25.451309 9131 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35045 every 8 connection(s)
I20260929 19:01:25.451793 9064 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708447806542-392-0/minicluster-data/master-0/data/info.pb
I20260929 19:01:25.452937 9132 sys_catalog.cc:263] Verifying existing consensus state
I20260929 19:01:25.453475 9132 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d19a32a47cb04612acc8e39a0386dc70: Bootstrap starting.
I20260929 19:01:25.460687 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUbnLHE/build/release/bin/kudu as pid 9064
I20260929 19:01:25.461035 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUbnLHE/build/release/bin/kudu
/tmp/dist-test-taskUbnLHE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708447806542-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708447806542-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708447806542-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708447806542-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:33311
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:35045
--webserver_port=44093
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708447806542-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.1
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708447806542-392-0/krb5kdc/oryx_127.0.98.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708447806542-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708447806542-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708447806542-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 19:01:25.461302 9132 log.cc:824] T 00000000000000000000000000000000 P d19a32a47cb04612acc8e39a0386dc70: Log is configured to *not* fsync() on all Append() calls
I20260929 19:01:25.462934 9132 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d19a32a47cb04612acc8e39a0386dc70: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 19:01:25.463209 9132 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d19a32a47cb04612acc8e39a0386dc70: Bootstrap complete.
I20260929 19:01:25.465459 9132 raft_consensus.cc:348] T 00000000000000000000000000000000 P d19a32a47cb04612acc8e39a0386dc70 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d19a32a47cb04612acc8e39a0386dc70" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35045 } }
I20260929 19:01:25.465668 9132 raft_consensus.cc:729] T 00000000000000000000000000000000 P d19a32a47cb04612acc8e39a0386dc70 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d19a32a47cb04612acc8e39a0386dc70, State: Initialized, Role: FOLLOWER
I20260929 19:01:25.465823 9132 consensus_queue.cc:260] T 00000000000000000000000000000000 P d19a32a47cb04612acc8e39a0386dc70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d19a32a47cb04612acc8e39a0386dc70" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35045 } }
I20260929 19:01:25.465898 9132 raft_consensus.cc:388] T 00000000000000000000000000000000 P d19a32a47cb04612acc8e39a0386dc70 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 19:01:25.465938 9132 raft_consensus.cc:482] T 00000000000000000000000000000000 P d19a32a47cb04612acc8e39a0386dc70 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 19:01:25.465981 9132 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d19a32a47cb04612acc8e39a0386dc70 [term 1 FOLLOWER]: Advancing to term 2
I20260929 19:01:25.466902 9132 raft_consensus.cc:504] T 00000000000000000000000000000000 P d19a32a47cb04612acc8e39a0386dc70 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d19a32a47cb04612acc8e39a0386dc70" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35045 } }
I20260929 19:01:25.467000 9132 leader_election.cc:302] T 00000000000000000000000000000000 P d19a32a47cb04612acc8e39a0386dc70 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d19a32a47cb04612acc8e39a0386dc70; no voters:
I20260929 19:01:25.467183 9132 leader_election.cc:288] T 00000000000000000000000000000000 P d19a32a47cb04612acc8e39a0386dc70 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 19:01:25.467214 9136 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d19a32a47cb04612acc8e39a0386dc70 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 19:01:25.467324 9136 raft_consensus.cc:686] T 00000000000000000000000000000000 P d19a32a47cb04612acc8e39a0386dc70 [term 2 LEADER]: Becoming Leader. State: Replica: d19a32a47cb04612acc8e39a0386dc70, State: Running, Role: LEADER
I20260929 19:01:25.467409 9136 consensus_queue.cc:237] T 00000000000000000000000000000000 P d19a32a47cb04612acc8e39a0386dc70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d19a32a47cb04612acc8e39a0386dc70" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35045 } }
I20260929 19:01:25.467437 9132 sys_catalog.cc:564] T 00000000000000000000000000000000 P d19a32a47cb04612acc8e39a0386dc70 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 19:01:25.467655 9136 sys_catalog.cc:455] T 00000000000000000000000000000000 P d19a32a47cb04612acc8e39a0386dc70 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "d19a32a47cb04612acc8e39a0386dc70" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d19a32a47cb04612acc8e39a0386dc70" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35045 } } }
I20260929 19:01:25.467718 9136 sys_catalog.cc:458] T 00000000000000000000000000000000 P d19a32a47cb04612acc8e39a0386dc70 [sys.catalog]: This master's current role is: LEADER
I20260929 19:01:25.467841 9136 sys_catalog.cc:455] T 00000000000000000000000000000000 P d19a32a47cb04612acc8e39a0386dc70 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d19a32a47cb04612acc8e39a0386dc70. Latest consensus state: current_term: 2 leader_uuid: "d19a32a47cb04612acc8e39a0386dc70" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d19a32a47cb04612acc8e39a0386dc70" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35045 } } }
I20260929 19:01:25.467904 9136 sys_catalog.cc:458] T 00000000000000000000000000000000 P d19a32a47cb04612acc8e39a0386dc70 [sys.catalog]: This master's current role is: LEADER
I20260929 19:01:25.468113 9140 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 19:01:25.468591 9140 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 19:01:25.468737 9140 catalog_manager.cc:1259] Loaded cluster ID: 930b2aeca3bf4076ba1fb8c667371fb0
I20260929 19:01:25.468779 9140 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 19:01:25.469735 9140 catalog_manager.cc:1504] Loading token signing keys...
I20260929 19:01:25.469916 9140 catalog_manager.cc:5879] T 00000000000000000000000000000000 P d19a32a47cb04612acc8e39a0386dc70: Loaded TSK: 0
I20260929 19:01:25.470122 9140 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 19:01:25.559829 9135 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:01:25.560034 9135 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:01:25.560070 9135 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:01:25.561218 9135 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:01:25.561269 9135 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:01:25.561312 9135 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 19:01:25.563063 9135 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708447806542-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708447806542-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708447806542-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:33311
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708447806542-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=44093
--tserver_master_addrs=127.0.98.62:35045
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9135
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708447806542-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 18:09:13 UTC on ccycloud.quasar-hpkalf.root.comops.site
build id 4680913
I20260929 19:01:25.563416 9135 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:01:25.563634 9135 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:01:25.569103 9159 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:01:25.569190 9161 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:01:25.569199 9158 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:01:25.569487 9135 server_base.cc:1034] running on GCE node
I20260929 19:01:25.569692 9135 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:01:25.569929 9135 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:01:25.571092 9135 hybrid_clock.cc:648] HybridClock initialized: now 1790708485571078 us; error 29 us; skew 500 ppm
Sep 29 19:01:25 dist-test-slave-lqkv krb5kdc[8551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708485, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:01:25.573693 9135 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260929 19:01:25.574184 9135 webserver.cc:460] Webserver started at http://127.0.98.1:44093/ using document root <none> and password file <none>
I20260929 19:01:25.574440 9135 fs_manager.cc:359] Metadata directory not provided
I20260929 19:01:25.574519 9135 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:01:25.576150 9135 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260929 19:01:25.576769 9168 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:25.576958 9135 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 19:01:25.577037 9135 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708447806542-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708447806542-392-0/minicluster-data/ts-0/wal
uuid: "b05badcafa5a4848a22be4b00c2f0bab"
format_stamp: "Formatted at 2026-09-29 19:01:24 on dist-test-slave-lqkv"
server_key: "e5be03752a4b90413e868cf856055f18"
server_key_iv: "031aac9a36978953127d1f39c0ea8075"
server_key_version: "encryptionkey@0"
I20260929 19:01:25.577324 9135 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708447806542-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708447806542-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708447806542-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:01:25.589509 9135 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:01:25.591636 9135 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:01:25.591773 9135 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:01:25.591989 9135 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:01:25.592283 9135 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:01:25.592337 9135 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:25.592377 9135 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:01:25.592404 9135 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:25.598462 9135 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33311
I20260929 19:01:25.598503 9281 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33311 every 8 connection(s)
I20260929 19:01:25.598801 9135 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708447806542-392-0/minicluster-data/ts-0/data/info.pb
I20260929 19:01:25.605890 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUbnLHE/build/release/bin/kudu as pid 9135
Sep 29 19:01:25 dist-test-slave-lqkv krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 19:01:25.606177 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUbnLHE/build/release/bin/kudu
/tmp/dist-test-taskUbnLHE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708447806542-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708447806542-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708447806542-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708447806542-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:34695
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:35045
--webserver_port=32911
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708447806542-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.2
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708447806542-392-0/krb5kdc/oryx_127.0.98.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708447806542-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708447806542-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708447806542-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 19:01:25 dist-test-slave-lqkv krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 19:01:25.607755 9284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:25.599590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35045 (local address 127.0.98.1:60083)
0929 19:01:25.599764 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:25.599802 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 19:01:25.600245 (+ 443us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:01:25.600683 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:01:25.600686 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:01:25.601306 (+ 620us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:01:25.601478 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:25.601483 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:25.603128 (+ 1645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:25.603129 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:25.603879 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:25.603880 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:25.604626 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:25.604626 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:25.604726 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:25.605267 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:01:25.605274 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:01:25.607538 (+ 2264us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35045: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":110,"threads_started":1}
W20260929 19:01:25.607846 9285 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:01:25.599663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60083 (local address 127.0.98.62:35045)
0929 19:01:25.599855 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:25.599863 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:25.599897 (+ 34us) server_negotiation.cc:407] Connection header received
0929 19:01:25.600446 (+ 549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:25.600453 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:25.600507 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:25.600649 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:01:25.601606 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:25.603024 (+ 1418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:25.603969 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:25.604542 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:25.604644 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:25.607675 (+ 3031us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:60083: BlockingRecv error: recv got EOF from 127.0.98.1:60083 (error 108)
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":81,"threads_started":1}
W20260929 19:01:25.608358 9282 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:35045 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:35045: Client connection negotiation failed: client connection to 127.0.98.62:35045: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 29 19:01:25 dist-test-slave-lqkv krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 19:01:25 dist-test-slave-lqkv krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 19:01:25.613276 9284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:25.608627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35045 (local address 127.0.98.1:49613)
0929 19:01:25.608678 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:25.608689 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 19:01:25.608734 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:01:25.608886 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:01:25.608887 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:01:25.608990 (+ 103us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:01:25.609029 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:25.609030 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:25.609822 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:25.609822 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:25.610297 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:25.610298 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:25.610779 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:25.610780 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:25.610824 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:25.611245 (+ 421us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:01:25.611249 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:01:25.613170 (+ 1921us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35045: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":35}
W20260929 19:01:25.613538 9285 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:01:25.608682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49613 (local address 127.0.98.62:35045)
0929 19:01:25.608726 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:25.608728 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:25.608737 (+ 9us) server_negotiation.cc:407] Connection header received
0929 19:01:25.608790 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:25.608791 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:25.608828 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:25.608879 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:01:25.609099 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:25.609747 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:25.610375 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:25.610702 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:25.610780 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:25.613460 (+ 2680us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:49613: BlockingRecv error: recv got EOF from 127.0.98.1:49613 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
Sep 29 19:01:25 dist-test-slave-lqkv krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 19:01:25 dist-test-slave-lqkv krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 19:01:25.618794 9284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:25.613554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35045 (local address 127.0.98.1:42355)
0929 19:01:25.613653 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:25.613666 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 19:01:25.613707 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:01:25.613974 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:01:25.613975 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:01:25.614130 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:01:25.614167 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:25.614168 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:25.615055 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:25.615056 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:25.615936 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:25.615938 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:25.616419 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:25.616420 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:25.616481 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:25.616942 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:01:25.616947 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:01:25.618731 (+ 1784us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35045: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":87,"thread_start_us":80,"threads_started":1}
W20260929 19:01:25.618867 9285 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:01:25.613711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42355 (local address 127.0.98.62:35045)
0929 19:01:25.613875 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:25.613877 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:25.613884 (+ 7us) server_negotiation.cc:407] Connection header received
0929 19:01:25.613903 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:25.613903 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:25.613912 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:25.613967 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:01:25.614258 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:25.614954 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:25.616030 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:25.616336 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:25.616404 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:25.618813 (+ 2409us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:42355: BlockingRecv error: recv got EOF from 127.0.98.1:42355 (error 108)
Metrics: {"server-negotiator.queue_time_us":147}
W20260929 19:01:25.619089 9282 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 19:01:25.689975 9286 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:01:25.690155 9286 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:01:25.690176 9286 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:01:25.691351 9286 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:01:25.691390 9286 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:01:25.691437 9286 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 19:01:25.692885 9286 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708447806542-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708447806542-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708447806542-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:34695
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708447806542-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=32911
--tserver_master_addrs=127.0.98.62:35045
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9286
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708447806542-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 18:09:13 UTC on ccycloud.quasar-hpkalf.root.comops.site
build id 4680913
I20260929 19:01:25.693117 9286 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:01:25.693341 9286 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:01:25.697952 9292 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:01:25.698025 9286 server_base.cc:1034] running on GCE node
W20260929 19:01:25.697981 9293 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:01:25.697963 9295 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:01:25.698378 9286 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:01:25.698603 9286 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:01:25.699744 9286 hybrid_clock.cc:648] HybridClock initialized: now 1790708485699733 us; error 29 us; skew 500 ppm
Sep 29 19:01:25 dist-test-slave-lqkv krb5kdc[8551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708485, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:01:25.702036 9286 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260929 19:01:25.702436 9286 webserver.cc:460] Webserver started at http://127.0.98.2:32911/ using document root <none> and password file <none>
I20260929 19:01:25.702661 9286 fs_manager.cc:359] Metadata directory not provided
I20260929 19:01:25.702740 9286 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:01:25.703969 9286 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260929 19:01:25.704514 9302 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:25.704677 9286 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 19:01:25.704769 9286 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708447806542-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708447806542-392-0/minicluster-data/ts-1/wal
uuid: "d02737d125f74c97adb08635f60a7ade"
format_stamp: "Formatted at 2026-09-29 19:01:25 on dist-test-slave-lqkv"
server_key: "d20fea4f680f1e7d2f0fffb7188121c1"
server_key_iv: "21d6cff243a65371c86e0b1b319afea0"
server_key_version: "encryptionkey@0"
I20260929 19:01:25.705117 9286 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708447806542-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708447806542-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708447806542-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:01:25.717269 9286 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:01:25.718856 9286 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:01:25.719007 9286 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:01:25.719211 9286 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:01:25.719477 9286 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:01:25.719527 9286 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:25.719558 9286 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:01:25.719584 9286 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:25.725709 9286 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34695
I20260929 19:01:25.725778 9415 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34695 every 8 connection(s)
I20260929 19:01:25.726087 9286 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708447806542-392-0/minicluster-data/ts-1/data/info.pb
Sep 29 19:01:25 dist-test-slave-lqkv krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 19:01:25.731748 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUbnLHE/build/release/bin/kudu as pid 9286
I20260929 19:01:25.732056 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUbnLHE/build/release/bin/kudu
/tmp/dist-test-taskUbnLHE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708447806542-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708447806542-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708447806542-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708447806542-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:44369
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:35045
--webserver_port=43945
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708447806542-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.3
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708447806542-392-0/krb5kdc/oryx_127.0.98.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708447806542-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708447806542-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708447806542-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 19:01:25 dist-test-slave-lqkv krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 19:01:25.733016 9418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:25.726700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35045 (local address 127.0.98.2:42023)
0929 19:01:25.726862 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:25.726903 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 19:01:25.727378 (+ 475us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:01:25.727610 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:01:25.727614 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:01:25.728118 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:01:25.728290 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:25.728294 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:25.729038 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:25.729039 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:25.729794 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:25.729797 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:25.730160 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:25.730161 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:25.730276 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:25.730753 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:01:25.730760 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:01:25.732894 (+ 2134us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35045: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":79,"threads_started":1}
W20260929 19:01:25.733062 9285 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:01:25.726751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42023 (local address 127.0.98.62:35045)
0929 19:01:25.726808 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:25.726811 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:25.726981 (+ 170us) server_negotiation.cc:407] Connection header received
0929 19:01:25.727481 (+ 500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:25.727483 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:25.727499 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:25.727550 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:01:25.728396 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:25.728948 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:25.729887 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:25.730078 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:25.730150 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:25.732981 (+ 2831us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:42023: BlockingRecv error: recv got EOF from 127.0.98.2:42023 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260929 19:01:25.733264 9416 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:35045 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:35045: Client connection negotiation failed: client connection to 127.0.98.62:35045: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 29 19:01:25 dist-test-slave-lqkv krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 19:01:25 dist-test-slave-lqkv krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260929 19:01:25.738431 9285 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:01:25.733529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58133 (local address 127.0.98.62:35045)
0929 19:01:25.733568 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:25.733570 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:25.733662 (+ 92us) server_negotiation.cc:407] Connection header received
0929 19:01:25.733700 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:25.733700 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:25.733708 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:25.733760 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:01:25.734056 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:25.734493 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:25.735536 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:25.735698 (+ 162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:25.735753 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:25.738395 (+ 2642us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:58133: BlockingRecv error: recv got EOF from 127.0.98.2:58133 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
I20260929 19:01:25.738428 9420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:25.733447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35045 (local address 127.0.98.2:58133)
0929 19:01:25.733573 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:25.733586 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 19:01:25.733648 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:01:25.733766 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:01:25.733767 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:01:25.733934 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:01:25.733993 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:25.733994 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:25.734565 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:25.734566 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:25.735456 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:25.735458 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:25.735785 (+ 327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:25.735787 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:25.735857 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:25.736215 (+ 358us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:01:25.736220 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:01:25.738326 (+ 2106us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35045: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":76,"threads_started":1}
Sep 29 19:01:25 dist-test-slave-lqkv krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 19:01:25 dist-test-slave-lqkv krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 19:01:25.743499 9420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:25.738922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35045 (local address 127.0.98.2:58333)
0929 19:01:25.738968 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:25.738977 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 19:01:25.739014 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:01:25.739131 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:01:25.739132 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:01:25.739286 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:01:25.739322 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:25.739324 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:25.739860 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:25.739861 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:25.740727 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:25.740730 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:25.741081 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:25.741082 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:25.741148 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:25.741534 (+ 386us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:01:25.741539 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:01:25.743428 (+ 1889us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35045: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":32}
W20260929 19:01:25.743548 9285 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:01:25.738983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58333 (local address 127.0.98.62:35045)
0929 19:01:25.739026 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:25.739027 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:25.739037 (+ 10us) server_negotiation.cc:407] Connection header received
0929 19:01:25.739057 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:25.739058 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:25.739064 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:25.739107 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:01:25.739394 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:25.739787 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:25.740846 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:25.741011 (+ 165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:25.741061 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:25.743488 (+ 2427us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:58333: BlockingRecv error: recv got EOF from 127.0.98.2:58333 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
W20260929 19:01:25.743621 9416 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 19:01:25.811959 9419 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:01:25.812160 9419 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:01:25.812196 9419 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:01:25.813375 9419 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:01:25.813434 9419 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:01:25.813477 9419 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 19:01:25.814862 9419 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708447806542-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708447806542-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708447806542-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:44369
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708447806542-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=43945
--tserver_master_addrs=127.0.98.62:35045
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9419
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708447806542-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 18:09:13 UTC on ccycloud.quasar-hpkalf.root.comops.site
build id 4680913
I20260929 19:01:25.815102 9419 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:01:25.815303 9419 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:01:25.819806 9425 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:01:25.819880 9426 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:01:25.819895 9419 server_base.cc:1034] running on GCE node
W20260929 19:01:25.819808 9428 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:01:25.820187 9419 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:01:25.820437 9419 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:01:25.821592 9419 hybrid_clock.cc:648] HybridClock initialized: now 1790708485821586 us; error 34 us; skew 500 ppm
Sep 29 19:01:25 dist-test-slave-lqkv krb5kdc[8551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708485, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:01:25.823825 9419 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260929 19:01:25.824160 9419 webserver.cc:460] Webserver started at http://127.0.98.3:43945/ using document root <none> and password file <none>
I20260929 19:01:25.824344 9419 fs_manager.cc:359] Metadata directory not provided
I20260929 19:01:25.824393 9419 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:01:25.825609 9419 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 19:01:25.826097 9435 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:25.826283 9419 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 19:01:25.826360 9419 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708447806542-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708447806542-392-0/minicluster-data/ts-2/wal
uuid: "8c8f98470f884957951c584ac1c8ef10"
format_stamp: "Formatted at 2026-09-29 19:01:25 on dist-test-slave-lqkv"
server_key: "23d7367736ed14a64f500bd933d0c9ea"
server_key_iv: "a37499165678784128ce56242efd19ed"
server_key_version: "encryptionkey@0"
I20260929 19:01:25.826613 9419 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708447806542-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708447806542-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708447806542-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:01:25.835173 9419 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:01:25.836774 9419 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:01:25.836936 9419 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:01:25.837095 9419 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:01:25.837306 9419 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:01:25.837333 9419 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:25.837353 9419 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:01:25.837365 9419 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:25.842774 9419 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44369
I20260929 19:01:25.842831 9548 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44369 every 8 connection(s)
I20260929 19:01:25.843147 9419 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790708447806542-392-0/minicluster-data/ts-2/data/info.pb
I20260929 19:01:25.846853 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUbnLHE/build/release/bin/kudu as pid 9419
Sep 29 19:01:25 dist-test-slave-lqkv krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 19:01:25 dist-test-slave-lqkv krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260929 19:01:25.850368 9285 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:01:25.843858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39397 (local address 127.0.98.62:35045)
0929 19:01:25.843934 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:25.843936 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:25.844076 (+ 140us) server_negotiation.cc:407] Connection header received
0929 19:01:25.844488 (+ 412us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:25.844489 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:25.844499 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:25.844551 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:01:25.845438 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:25.845899 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:25.846834 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:25.847153 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:25.847225 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:25.850308 (+ 3083us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:39397: BlockingRecv error: recv got EOF from 127.0.98.3:39397 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 19:01:25.850390 9551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:25.843822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35045 (local address 127.0.98.3:39397)
0929 19:01:25.843969 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:25.843998 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0929 19:01:25.844398 (+ 400us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:01:25.844597 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:01:25.844599 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:01:25.845190 (+ 591us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:01:25.845366 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:25.845370 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:25.845970 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:25.845971 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:25.846737 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:25.846738 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:25.847244 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:25.847245 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:25.847398 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:25.847868 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:01:25.847875 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:01:25.850212 (+ 2337us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35045: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":88,"threads_started":1}
W20260929 19:01:25.850657 9549 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:35045 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:35045: Client connection negotiation failed: client connection to 127.0.98.62:35045: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 29 19:01:25 dist-test-slave-lqkv krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 19:01:25 dist-test-slave-lqkv krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 19:01:25.855145 9553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:25.847415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57536 (local address 127.0.98.62:35045)
0929 19:01:25.847519 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:25.847521 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:25.847540 (+ 19us) server_negotiation.cc:407] Connection header received
0929 19:01:25.847561 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:25.847562 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:25.847571 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:25.847625 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:01:25.848186 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:25.848922 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:25.849901 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:25.850199 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:25.850265 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:25.851793 (+ 1528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:01:25.851797 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:01:25.851797 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:01:25.851806 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:01:25.853682 (+ 1876us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:25.854073 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:25.854076 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:25.854076 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:25.854188 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:25.854424 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:25.854425 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:25.854425 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:25.854707 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:01:25.854860 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0929 19:01:25.854975 (+ 115us) server_negotiation.cc:299] Negotiation successful
0929 19:01:25.855052 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89,"thread_start_us":57,"threads_started":1}
I20260929 19:01:25.855548 9551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:25.850893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35045 (local address 127.0.98.3:55707)
0929 19:01:25.850995 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:25.851008 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 19:01:25.851045 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:01:25.851162 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:01:25.851162 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:01:25.851263 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:01:25.851306 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:25.851307 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:25.852094 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:25.852095 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:25.852956 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:25.852959 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:25.853306 (+ 347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:25.853307 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:25.853374 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:25.853685 (+ 311us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:01:25.853690 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:01:25.855480 (+ 1790us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35045: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":87}
W20260929 19:01:25.855616 9285 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:01:25.850957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55707 (local address 127.0.98.62:35045)
0929 19:01:25.850992 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:25.850994 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:25.851038 (+ 44us) server_negotiation.cc:407] Connection header received
0929 19:01:25.851082 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:25.851084 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:25.851092 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:25.851159 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:01:25.851374 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:25.852022 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:25.853033 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:25.853213 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:25.853282 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:25.855561 (+ 2279us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:55707: BlockingRecv error: recv got EOF from 127.0.98.3:55707 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
Sep 29 19:01:25 dist-test-slave-lqkv krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 19:01:25 dist-test-slave-lqkv krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 19:01:25.859403 9551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:25.855776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35045 (local address 127.0.98.3:37377)
0929 19:01:25.855876 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:25.855885 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 19:01:25.855928 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:01:25.856071 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:01:25.856072 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:01:25.856169 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:01:25.856192 (+ 23us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:25.856193 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:25.856703 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:25.856704 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:25.857236 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:25.857238 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:25.857523 (+ 285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:25.857524 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:25.857557 (+ 33us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:25.857938 (+ 381us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:01:25.857943 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:01:25.859333 (+ 1390us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35045: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":65,"threads_started":1}
W20260929 19:01:25.859431 9285 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:01:25.855841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37377 (local address 127.0.98.62:35045)
0929 19:01:25.855876 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:25.855878 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:25.855917 (+ 39us) server_negotiation.cc:407] Connection header received
0929 19:01:25.855986 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:25.855987 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:25.855995 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:25.856048 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:01:25.856244 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:25.856649 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:25.857303 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:25.857462 (+ 159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:25.857512 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:25.859397 (+ 1885us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:37377: BlockingRecv error: recv got EOF from 127.0.98.3:37377 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260929 19:01:25.859519 9549 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 29 19:01:26 dist-test-slave-lqkv krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 19:01:26 dist-test-slave-lqkv krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 19:01:26.623577 9555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:26.619439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35045 (local address 127.0.98.1:49405)
0929 19:01:26.619575 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:26.619585 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 19:01:26.619628 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:01:26.619755 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:01:26.619756 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:01:26.619911 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:01:26.619956 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:26.619957 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:26.620548 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:26.620549 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:26.621164 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:26.621166 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:26.621490 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:26.621491 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:26.621525 (+ 34us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:26.621948 (+ 423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:01:26.621952 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:01:26.623517 (+ 1565us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35045: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":78,"threads_started":1}
W20260929 19:01:26.623706 9556 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:01:26.619491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49405 (local address 127.0.98.62:35045)
0929 19:01:26.619607 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:26.619609 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:26.619624 (+ 15us) server_negotiation.cc:407] Connection header received
0929 19:01:26.619680 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:26.619681 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:26.619689 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:26.619736 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:01:26.620016 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:26.620487 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:26.621232 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:26.621434 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:26.621480 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:26.623606 (+ 2126us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:49405: BlockingRecv error: recv got EOF from 127.0.98.1:49405 (error 108)
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":70,"threads_started":1}
Sep 29 19:01:26 dist-test-slave-lqkv krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 19:01:26 dist-test-slave-lqkv krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 19:01:26.748939 9557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:26.743926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35045 (local address 127.0.98.2:54765)
0929 19:01:26.744112 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:26.744123 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 19:01:26.744222 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:01:26.744371 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:01:26.744372 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:01:26.744550 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:01:26.744612 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:26.744614 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:26.745498 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:26.745499 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:26.746077 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:26.746079 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:26.746445 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:26.746446 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:26.746523 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:26.747059 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:01:26.747066 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:01:26.748885 (+ 1819us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35045: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":82,"threads_started":1}
W20260929 19:01:26.749085 9556 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:01:26.744003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54765 (local address 127.0.98.62:35045)
0929 19:01:26.744100 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:26.744103 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:26.744205 (+ 102us) server_negotiation.cc:407] Connection header received
0929 19:01:26.744278 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:26.744279 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:26.744291 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:26.744366 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:01:26.744706 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:26.745415 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:26.746182 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:26.746376 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:26.746447 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:26.748989 (+ 2542us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:54765: BlockingRecv error: recv got EOF from 127.0.98.2:54765 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 29 19:01:26 dist-test-slave-lqkv krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 19:01:26 dist-test-slave-lqkv krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 19:01:26.864377 9558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:26.859829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35045 (local address 127.0.98.3:43121)
0929 19:01:26.859954 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:26.859965 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 19:01:26.860028 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:01:26.860142 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:01:26.860142 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:01:26.860289 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:01:26.860335 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:26.860336 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:26.860960 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:26.860961 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:26.861480 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:26.861482 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:26.861781 (+ 299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:26.861782 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:26.861817 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:26.862306 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:01:26.862312 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:01:26.864281 (+ 1969us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35045: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":83,"threads_started":1}
W20260929 19:01:26.864435 9556 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:01:26.859899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43121 (local address 127.0.98.62:35045)
0929 19:01:26.859959 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:26.859961 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:26.860030 (+ 69us) server_negotiation.cc:407] Connection header received
0929 19:01:26.860069 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:26.860070 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:26.860080 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:26.860127 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:01:26.860396 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:26.860900 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:26.861550 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:26.861723 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:26.861770 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:26.864369 (+ 2599us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:43121: BlockingRecv error: recv got EOF from 127.0.98.3:43121 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 29 19:01:27 dist-test-slave-lqkv krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 19:01:27 dist-test-slave-lqkv krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 19:01:27.628580 9559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:27.624079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35045 (local address 127.0.98.1:44451)
0929 19:01:27.624210 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:27.624223 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 19:01:27.624282 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:01:27.624462 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:01:27.624463 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:01:27.624635 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:01:27.624682 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:27.624685 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:27.625403 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:27.625404 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:27.625937 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:27.625938 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:27.626341 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:27.626342 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:27.626399 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:27.626868 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:01:27.626872 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:01:27.628525 (+ 1653us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35045: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":70,"threads_started":1}
W20260929 19:01:27.628753 9560 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:01:27.624165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44451 (local address 127.0.98.62:35045)
0929 19:01:27.624312 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:27.624314 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:27.624323 (+ 9us) server_negotiation.cc:407] Connection header received
0929 19:01:27.624345 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:27.624346 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:27.624358 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:27.624441 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:01:27.624762 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:27.625320 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:27.626018 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:27.626243 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:27.626339 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:27.628613 (+ 2274us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:44451: BlockingRecv error: recv got EOF from 127.0.98.1:44451 (error 108)
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":81,"threads_started":1}
Sep 29 19:01:27 dist-test-slave-lqkv krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 19:01:27 dist-test-slave-lqkv krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 19:01:27.753710 9561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:27.749457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35045 (local address 127.0.98.2:50215)
0929 19:01:27.749596 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:27.749610 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 19:01:27.749664 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:01:27.749789 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:01:27.749790 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:01:27.749940 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:01:27.749986 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:27.749987 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:27.750584 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:27.750585 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:27.751095 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:27.751096 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:27.751387 (+ 291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:27.751388 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:27.751421 (+ 33us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:27.751839 (+ 418us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:01:27.751843 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:01:27.753658 (+ 1815us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35045: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":89,"threads_started":1}
W20260929 19:01:27.753821 9560 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:01:27.749514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50215 (local address 127.0.98.62:35045)
0929 19:01:27.749584 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:27.749586 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:27.749673 (+ 87us) server_negotiation.cc:407] Connection header received
0929 19:01:27.749710 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:27.749710 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:27.749719 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:27.749786 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:01:27.750044 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:27.750518 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:27.751159 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:27.751331 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:27.751378 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:27.753745 (+ 2367us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:50215: BlockingRecv error: recv got EOF from 127.0.98.2:50215 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Sep 29 19:01:27 dist-test-slave-lqkv krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 19:01:27 dist-test-slave-lqkv krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 19:01:27.869582 9562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:27.864888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35045 (local address 127.0.98.3:59959)
0929 19:01:27.865008 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:27.865041 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0929 19:01:27.865100 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:01:27.865240 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:01:27.865241 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:01:27.865414 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:01:27.865458 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:27.865459 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:27.866114 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:27.866115 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:27.866622 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:27.866623 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:27.866975 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:27.866976 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:27.867036 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:27.867555 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:01:27.867560 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:01:27.869483 (+ 1923us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35045: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":78,"threads_started":1}
W20260929 19:01:27.869620 9560 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:01:27.865001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59959 (local address 127.0.98.62:35045)
0929 19:01:27.865071 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:27.865073 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:27.865082 (+ 9us) server_negotiation.cc:407] Connection header received
0929 19:01:27.865173 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:27.865173 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:27.865187 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:27.865237 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:01:27.865542 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:27.866030 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:27.866709 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:27.866894 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:27.866972 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:27.869555 (+ 2583us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:59959: BlockingRecv error: recv got EOF from 127.0.98.3:59959 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 29 19:01:28 dist-test-slave-lqkv krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 19:01:28 dist-test-slave-lqkv krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 19:01:28.633795 9563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:28.629049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35045 (local address 127.0.98.1:46635)
0929 19:01:28.629238 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:28.629248 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 19:01:28.629342 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:01:28.629516 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:01:28.629517 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:01:28.629701 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:01:28.629779 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:28.629781 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:28.630496 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:28.630497 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:28.631012 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:28.631013 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:28.631323 (+ 310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:28.631324 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:28.631362 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:28.631859 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:01:28.631864 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:01:28.633724 (+ 1860us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35045: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":115,"threads_started":1}
W20260929 19:01:28.633904 9564 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:01:28.629146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46635 (local address 127.0.98.62:35045)
0929 19:01:28.629280 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:28.629282 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:28.629317 (+ 35us) server_negotiation.cc:407] Connection header received
0929 19:01:28.629392 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:28.629392 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:28.629402 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:28.629457 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:01:28.629847 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:28.630419 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:28.631076 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:28.631265 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:28.631313 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:28.633803 (+ 2490us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:46635: BlockingRecv error: recv got EOF from 127.0.98.1:46635 (error 108)
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":58,"threads_started":1}
Sep 29 19:01:28 dist-test-slave-lqkv krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 19:01:28 dist-test-slave-lqkv krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 19:01:28.758388 9565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:28.754256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35045 (local address 127.0.98.2:46061)
0929 19:01:28.754386 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:28.754398 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 19:01:28.754459 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:01:28.754578 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:01:28.754579 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:01:28.754780 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:01:28.754825 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:28.754827 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:28.755423 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:28.755423 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:28.755936 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:28.755937 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:28.756240 (+ 303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:28.756240 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:28.756285 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:28.756698 (+ 413us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:01:28.756702 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:01:28.758335 (+ 1633us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35045: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":61,"threads_started":1}
W20260929 19:01:28.758508 9564 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:01:28.754320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46061 (local address 127.0.98.62:35045)
0929 19:01:28.754385 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:28.754387 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:28.754460 (+ 73us) server_negotiation.cc:407] Connection header received
0929 19:01:28.754501 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:28.754501 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:28.754511 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:28.754560 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:01:28.754885 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:28.755360 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:28.756002 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:28.756183 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:28.756229 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:28.758421 (+ 2192us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:46061: BlockingRecv error: recv got EOF from 127.0.98.2:46061 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 29 19:01:28 dist-test-slave-lqkv krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 19:01:28 dist-test-slave-lqkv krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 19:01:28.874492 9566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:28.870010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35045 (local address 127.0.98.3:60109)
0929 19:01:28.870168 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:28.870181 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 19:01:28.870262 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:01:28.870429 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:01:28.870430 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:01:28.870605 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:01:28.870650 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:28.870652 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:28.871390 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:28.871391 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:28.871921 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:28.871922 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:28.872284 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:28.872285 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:28.872343 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:28.872784 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:01:28.872788 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:01:28.874437 (+ 1649us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35045: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":80,"threads_started":1}
W20260929 19:01:28.874588 9564 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:01:28.870086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60109 (local address 127.0.98.62:35045)
0929 19:01:28.870154 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:28.870157 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:28.870245 (+ 88us) server_negotiation.cc:407] Connection header received
0929 19:01:28.870338 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:28.870339 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:28.870353 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:28.870422 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:01:28.870729 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:28.871295 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:28.872004 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:28.872200 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:28.872284 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:28.874521 (+ 2237us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:60109: BlockingRecv error: recv got EOF from 127.0.98.3:60109 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 29 19:01:29 dist-test-slave-lqkv krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 19:01:29 dist-test-slave-lqkv krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 19:01:29.638758 9567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:29.634212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35045 (local address 127.0.98.1:54303)
0929 19:01:29.634358 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:29.634370 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 19:01:29.634426 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:01:29.634569 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:01:29.634569 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:01:29.634722 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:01:29.634770 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:29.634771 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:29.635464 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:29.635465 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:29.635985 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:29.635987 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:29.636324 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:29.636324 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:29.636361 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:29.636890 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:01:29.636895 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:01:29.638666 (+ 1771us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35045: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":89,"threads_started":1}
W20260929 19:01:29.638818 9568 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:01:29.634278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54303 (local address 127.0.98.62:35045)
0929 19:01:29.634408 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:29.634409 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:29.634418 (+ 9us) server_negotiation.cc:407] Connection header received
0929 19:01:29.634491 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:29.634492 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:29.634501 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:29.634550 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:01:29.634880 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:29.635392 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:29.636063 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:29.636255 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:29.636312 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:29.638750 (+ 2438us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:54303: BlockingRecv error: recv got EOF from 127.0.98.1:54303 (error 108)
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":77,"threads_started":1}
Sep 29 19:01:29 dist-test-slave-lqkv krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 19:01:29 dist-test-slave-lqkv krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 19:01:29.766834 9569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:29.758840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35045 (local address 127.0.98.2:33309)
0929 19:01:29.759022 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:29.759034 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 19:01:29.759115 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:01:29.759240 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:01:29.759241 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:01:29.759413 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:01:29.759456 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:29.759458 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:29.760115 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:29.760116 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:29.760629 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:29.760631 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:29.761058 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:29.761059 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:29.761112 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:29.765006 (+ 3894us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:01:29.765010 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:01:29.766766 (+ 1756us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35045: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":100,"threads_started":1}
W20260929 19:01:29.766924 9568 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:01:29.758923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33309 (local address 127.0.98.62:35045)
0929 19:01:29.759009 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:29.759011 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:29.759099 (+ 88us) server_negotiation.cc:407] Connection header received
0929 19:01:29.759168 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:29.759169 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:29.759181 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:29.759228 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:01:29.759549 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:29.760035 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:29.760713 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:29.760977 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:29.761070 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:29.766857 (+ 5787us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:33309: BlockingRecv error: recv got EOF from 127.0.98.2:33309 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 29 19:01:29 dist-test-slave-lqkv krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 19:01:29 dist-test-slave-lqkv krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 19:01:29.879339 9570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:29.874919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35045 (local address 127.0.98.3:41815)
0929 19:01:29.875033 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:29.875046 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 19:01:29.875112 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:01:29.875311 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:01:29.875312 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:01:29.875511 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:01:29.875568 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:29.875570 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:29.876201 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:29.876202 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:29.876706 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:29.876708 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:29.877031 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:29.877032 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:29.877081 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:29.877553 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:01:29.877557 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:01:29.879248 (+ 1691us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35045: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":68,"threads_started":1}
W20260929 19:01:29.879385 9568 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:01:29.875007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41815 (local address 127.0.98.62:35045)
0929 19:01:29.875091 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:29.875093 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:29.875102 (+ 9us) server_negotiation.cc:407] Connection header received
0929 19:01:29.875204 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:29.875204 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:29.875229 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:29.875288 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:01:29.875635 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:29.876137 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:29.876770 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:29.876963 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:29.877012 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:29.879337 (+ 2325us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:41815: BlockingRecv error: recv got EOF from 127.0.98.3:41815 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Sep 29 19:01:30 dist-test-slave-lqkv krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 19:01:30 dist-test-slave-lqkv krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 19:01:30.643744 9571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:30.639245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35045 (local address 127.0.98.1:32775)
0929 19:01:30.639380 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:30.639390 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 19:01:30.639433 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:01:30.639586 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:01:30.639587 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:01:30.639762 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:01:30.639809 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:30.639810 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:30.640491 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:30.640492 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:30.641119 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:30.641121 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:30.641437 (+ 316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:30.641438 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:30.641473 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:30.641937 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:01:30.641941 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:01:30.643643 (+ 1702us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35045: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":83,"threads_started":1}
W20260929 19:01:30.643779 9572 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:01:30.639284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:32775 (local address 127.0.98.62:35045)
0929 19:01:30.639432 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:30.639434 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:30.639442 (+ 8us) server_negotiation.cc:407] Connection header received
0929 19:01:30.639492 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:30.639493 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:30.639503 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:30.639571 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:01:30.639898 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:30.640391 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:30.641185 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:30.641376 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:30.641430 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:30.643711 (+ 2281us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:32775: BlockingRecv error: recv got EOF from 127.0.98.1:32775 (error 108)
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":94,"threads_started":1}
Sep 29 19:01:30 dist-test-slave-lqkv krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 19:01:30 dist-test-slave-lqkv krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 19:01:30.771696 9573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:30.767327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35045 (local address 127.0.98.2:54815)
0929 19:01:30.767462 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:30.767480 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 19:01:30.767527 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:01:30.767628 (+ 101us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:01:30.767629 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:01:30.767827 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:01:30.767870 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:30.767871 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:30.768580 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:30.768581 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:30.769157 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:30.769159 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:30.769451 (+ 292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:30.769452 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:30.769489 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:30.769931 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:01:30.769935 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:01:30.771641 (+ 1706us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35045: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":67,"threads_started":1}
W20260929 19:01:30.771818 9572 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:01:30.767475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54815 (local address 127.0.98.62:35045)
0929 19:01:30.767531 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:30.767535 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:30.767547 (+ 12us) server_negotiation.cc:407] Connection header received
0929 19:01:30.767568 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:30.767569 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:30.767578 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:30.767625 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:01:30.767930 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:30.768506 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:30.769221 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:30.769393 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:30.769441 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:30.771737 (+ 2296us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:54815: BlockingRecv error: recv got EOF from 127.0.98.2:54815 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Sep 29 19:01:30 dist-test-slave-lqkv krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 19:01:30 dist-test-slave-lqkv krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 19:01:30.884292 9574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:30.879771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35045 (local address 127.0.98.3:59171)
0929 19:01:30.879942 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:30.879953 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 19:01:30.880016 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:01:30.880144 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:01:30.880144 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:01:30.880313 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:01:30.880358 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:30.880360 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:30.881159 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:30.881160 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:30.881728 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:30.881730 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:30.882096 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:30.882097 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:30.882156 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:30.882569 (+ 413us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:01:30.882573 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:01:30.884230 (+ 1657us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35045: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":101,"threads_started":1}
W20260929 19:01:30.884450 9572 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:01:30.879840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59171 (local address 127.0.98.62:35045)
0929 19:01:30.879923 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:30.879924 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:30.880008 (+ 84us) server_negotiation.cc:407] Connection header received
0929 19:01:30.880075 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:30.880076 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:30.880090 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:30.880140 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:01:30.880433 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:30.881073 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:30.881814 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:30.882021 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:30.882098 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:30.884363 (+ 2265us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:59171: BlockingRecv error: recv got EOF from 127.0.98.3:59171 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 29 19:01:31 dist-test-slave-lqkv krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 19:01:31 dist-test-slave-lqkv krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 19:01:31.648442 9575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:31.644365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35045 (local address 127.0.98.1:40937)
0929 19:01:31.644492 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:31.644507 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 19:01:31.644552 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:01:31.644689 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:01:31.644690 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:01:31.644858 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:01:31.644904 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:31.644905 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:31.645507 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:31.645507 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:31.646010 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:31.646012 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:31.646302 (+ 290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:31.646303 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:31.646340 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:31.646793 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:01:31.646797 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:01:31.648386 (+ 1589us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35045: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":80,"threads_started":1}
W20260929 19:01:31.648579 9576 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:01:31.644449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40937 (local address 127.0.98.62:35045)
0929 19:01:31.644580 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:31.644581 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:31.644588 (+ 7us) server_negotiation.cc:407] Connection header received
0929 19:01:31.644607 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:31.644608 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:31.644618 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:31.644675 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:01:31.644967 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:31.645444 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:31.646074 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:31.646243 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:31.646290 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:31.648490 (+ 2200us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:40937: BlockingRecv error: recv got EOF from 127.0.98.1:40937 (error 108)
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":79,"threads_started":1}
Sep 29 19:01:31 dist-test-slave-lqkv krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 19:01:31 dist-test-slave-lqkv krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 19:01:31.776386 9577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:31.772118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35045 (local address 127.0.98.2:42315)
0929 19:01:31.772217 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:31.772229 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 19:01:31.772292 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:01:31.772430 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:01:31.772431 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:01:31.772574 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:01:31.772620 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:31.772621 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:31.773365 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:31.773365 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:31.773878 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:31.773880 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:31.774166 (+ 286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:31.774166 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:31.774201 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:31.774668 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:01:31.774673 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:01:31.776328 (+ 1655us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35045: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":79,"thread_start_us":47,"threads_started":1}
W20260929 19:01:31.776507 9576 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:01:31.772178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42315 (local address 127.0.98.62:35045)
0929 19:01:31.772252 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:31.772254 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:31.772277 (+ 23us) server_negotiation.cc:407] Connection header received
0929 19:01:31.772353 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:31.772354 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:31.772364 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:31.772410 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:01:31.772673 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:31.773305 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:31.773941 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:31.774108 (+ 167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:31.774154 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:31.776410 (+ 2256us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:42315: BlockingRecv error: recv got EOF from 127.0.98.2:42315 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 29 19:01:31 dist-test-slave-lqkv krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 19:01:31 dist-test-slave-lqkv krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 19:01:31.888903 9578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:31.884725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35045 (local address 127.0.98.3:49121)
0929 19:01:31.884892 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:31.884903 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 19:01:31.884958 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:01:31.885083 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:01:31.885084 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:01:31.885238 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:01:31.885282 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:31.885283 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:31.885876 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:31.885876 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:31.886379 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:31.886380 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:31.886686 (+ 306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:31.886687 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:31.886722 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:31.887166 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:01:31.887170 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:01:31.888776 (+ 1606us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35045: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":105,"threads_started":1}
W20260929 19:01:31.888950 9576 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:01:31.884770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49121 (local address 127.0.98.62:35045)
0929 19:01:31.884861 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:31.884863 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:31.884961 (+ 98us) server_negotiation.cc:407] Connection header received
0929 19:01:31.884997 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:31.884998 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:31.885020 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:31.885067 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:01:31.885338 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:31.885810 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:31.886444 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:31.886608 (+ 164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:31.886675 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:31.888905 (+ 2230us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:49121: BlockingRecv error: recv got EOF from 127.0.98.3:49121 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 29 19:01:32 dist-test-slave-lqkv krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 19:01:32 dist-test-slave-lqkv krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 19:01:32.653193 9579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:32.648946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35045 (local address 127.0.98.1:60785)
0929 19:01:32.649061 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:32.649078 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 19:01:32.649123 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:01:32.649343 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:01:32.649344 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:01:32.649492 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:01:32.649538 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:32.649540 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:32.650157 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:32.650157 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:32.650661 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:32.650662 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:32.650969 (+ 307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:32.650970 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:32.651006 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:32.651426 (+ 420us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:01:32.651430 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:01:32.653138 (+ 1708us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35045: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":69,"threads_started":1}
W20260929 19:01:32.653316 9580 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:01:32.649070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60785 (local address 127.0.98.62:35045)
0929 19:01:32.649214 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:32.649218 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:32.649239 (+ 21us) server_negotiation.cc:407] Connection header received
0929 19:01:32.649259 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:32.649259 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:32.649269 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:32.649331 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:01:32.649602 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:32.650091 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:32.650729 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:32.650909 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:32.650959 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:32.653234 (+ 2275us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:60785: BlockingRecv error: recv got EOF from 127.0.98.1:60785 (error 108)
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":90,"threads_started":1}
Sep 29 19:01:32 dist-test-slave-lqkv krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 19:01:32 dist-test-slave-lqkv krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 19:01:32.781479 9581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:32.776923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35045 (local address 127.0.98.2:32771)
0929 19:01:32.777074 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:32.777084 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 19:01:32.777144 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:01:32.777270 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:01:32.777271 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:01:32.777435 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:01:32.777478 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:32.777479 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:32.778104 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:32.778105 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:32.778624 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:32.778626 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:32.778945 (+ 319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:32.778946 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:32.778983 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:32.779400 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:01:32.779404 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:01:32.781423 (+ 2019us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35045: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":89,"threads_started":1}
W20260929 19:01:32.781610 9580 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:01:32.776950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:32771 (local address 127.0.98.62:35045)
0929 19:01:32.776996 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:32.776998 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:32.777145 (+ 147us) server_negotiation.cc:407] Connection header received
0929 19:01:32.777187 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:32.777188 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:32.777200 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:32.777249 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:01:32.777535 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:32.778041 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:32.778713 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:32.778887 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:32.778935 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:32.781532 (+ 2597us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:32771: BlockingRecv error: recv got EOF from 127.0.98.2:32771 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
Sep 29 19:01:32 dist-test-slave-lqkv krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 19:01:32 dist-test-slave-lqkv krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 19:01:32.894310 9582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:32.889417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35045 (local address 127.0.98.3:51057)
0929 19:01:32.889596 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:32.889606 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 19:01:32.889662 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:01:32.889834 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:01:32.889835 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:01:32.890096 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:01:32.890156 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:32.890158 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:32.890860 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:32.890861 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:32.891383 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:32.891385 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:32.891725 (+ 340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:32.891726 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:32.891769 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:32.892344 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:01:32.892349 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:01:32.894234 (+ 1885us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35045: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":105,"threads_started":1}
W20260929 19:01:32.894373 9580 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:01:32.889500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51057 (local address 127.0.98.62:35045)
0929 19:01:32.889572 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:32.889573 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:32.889709 (+ 136us) server_negotiation.cc:407] Connection header received
0929 19:01:32.889738 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:32.889739 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:32.889755 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:32.889805 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:01:32.890270 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:32.890785 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:32.891471 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:32.891653 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:32.891708 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:32.894327 (+ 2619us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:51057: BlockingRecv error: recv got EOF from 127.0.98.3:51057 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 29 19:01:33 dist-test-slave-lqkv krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 19:01:33 dist-test-slave-lqkv krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 19:01:33.658205 9583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:33.653657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35045 (local address 127.0.98.1:35229)
0929 19:01:33.653787 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:33.653805 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 19:01:33.653853 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:01:33.654023 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:01:33.654024 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:01:33.654189 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:01:33.654235 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:33.654236 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:33.654875 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:33.654876 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:33.655406 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:33.655407 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:33.655711 (+ 304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:33.655711 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:33.655748 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:33.656244 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:01:33.656248 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:01:33.658139 (+ 1891us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35045: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":85,"threads_started":1}
W20260929 19:01:33.658303 9584 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:01:33.653761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35229 (local address 127.0.98.62:35045)
0929 19:01:33.653882 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:33.653883 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:33.653890 (+ 7us) server_negotiation.cc:407] Connection header received
0929 19:01:33.653909 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:33.653910 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:33.653919 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:33.653978 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:01:33.654304 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:33.654809 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:33.655473 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:33.655651 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:33.655698 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:33.658225 (+ 2527us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:35229: BlockingRecv error: recv got EOF from 127.0.98.1:35229 (error 108)
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":70,"threads_started":1}
Sep 29 19:01:33 dist-test-slave-lqkv krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 19:01:33 dist-test-slave-lqkv krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 19:01:33.786531 9585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:33.781864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35045 (local address 127.0.98.2:45273)
0929 19:01:33.781986 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:33.782005 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 19:01:33.782059 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:01:33.782191 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:01:33.782192 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:01:33.782352 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:01:33.782396 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:33.782397 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:33.783236 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:33.783237 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:33.783773 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:33.783775 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:33.784090 (+ 315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:33.784091 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:33.784131 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:33.784559 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:01:33.784564 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:01:33.786429 (+ 1865us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35045: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":68,"threads_started":1}
W20260929 19:01:33.786567 9584 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:01:33.781948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45273 (local address 127.0.98.62:35045)
0929 19:01:33.782029 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:33.782031 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:33.782041 (+ 10us) server_negotiation.cc:407] Connection header received
0929 19:01:33.782114 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:33.782114 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:33.782125 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:33.782170 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:01:33.782478 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:33.783163 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:33.783847 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:33.784028 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:33.784078 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:33.786505 (+ 2427us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:45273: BlockingRecv error: recv got EOF from 127.0.98.2:45273 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 29 19:01:33 dist-test-slave-lqkv krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 19:01:33 dist-test-slave-lqkv krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 19:01:33.898962 9586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:33.894767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35045 (local address 127.0.98.3:49251)
0929 19:01:33.894902 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:33.894911 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 19:01:33.894960 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:01:33.895075 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:01:33.895076 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:01:33.895227 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:01:33.895273 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:33.895274 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:33.895866 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:33.895866 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:33.896349 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:33.896350 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:33.896687 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:33.896688 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:33.896751 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:33.897226 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:01:33.897231 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:01:33.898906 (+ 1675us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35045: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":80,"threads_started":1}
W20260929 19:01:33.899062 9584 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:01:33.894843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49251 (local address 127.0.98.62:35045)
0929 19:01:33.894898 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:33.894901 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:33.894949 (+ 48us) server_negotiation.cc:407] Connection header received
0929 19:01:33.894999 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:33.895000 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:33.895010 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:33.895057 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:01:33.895331 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:33.895800 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:33.896431 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:33.896600 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:33.896649 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:33.898992 (+ 2343us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:49251: BlockingRecv error: recv got EOF from 127.0.98.3:49251 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Sep 29 19:01:34 dist-test-slave-lqkv krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 19:01:34 dist-test-slave-lqkv krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 19:01:34.663228 9587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:34.658635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35045 (local address 127.0.98.1:44197)
0929 19:01:34.658757 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:34.658766 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 19:01:34.658806 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:01:34.659015 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:01:34.659016 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:01:34.659166 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:01:34.659213 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:34.659215 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:34.659850 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:34.659851 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:34.660349 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:34.660351 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:34.660643 (+ 292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:34.660643 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:34.660680 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:34.661215 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:01:34.661227 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:01:34.663173 (+ 1946us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35045: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":53,"threads_started":1}
W20260929 19:01:34.663332 9588 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:01:34.658749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44197 (local address 127.0.98.62:35045)
0929 19:01:34.658889 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:34.658891 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:34.658900 (+ 9us) server_negotiation.cc:407] Connection header received
0929 19:01:34.658927 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:34.658928 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:34.658937 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:34.658998 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:01:34.659286 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:34.659789 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:34.660412 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:34.660583 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:34.660629 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:34.663261 (+ 2632us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:44197: BlockingRecv error: recv got EOF from 127.0.98.1:44197 (error 108)
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":95,"threads_started":1}
Sep 29 19:01:34 dist-test-slave-lqkv krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 19:01:34 dist-test-slave-lqkv krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 19:01:34.795109 9589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:34.787054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35045 (local address 127.0.98.2:52573)
0929 19:01:34.787217 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:34.787229 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 19:01:34.787318 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:01:34.787465 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:01:34.787466 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:01:34.787640 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:01:34.787684 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:34.787686 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:34.788379 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:34.788380 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:34.788940 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:34.788942 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:34.789332 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:34.789334 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:34.789397 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:34.793071 (+ 3674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:01:34.793084 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:01:34.795048 (+ 1964us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35045: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":92,"threads_started":1}
W20260929 19:01:34.795327 9588 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:01:34.787136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52573 (local address 127.0.98.62:35045)
0929 19:01:34.787211 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:34.787213 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:34.787305 (+ 92us) server_negotiation.cc:407] Connection header received
0929 19:01:34.787381 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:34.787382 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:34.787399 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:34.787459 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:01:34.787763 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:34.788296 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:34.789026 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:34.789223 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:34.789332 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:34.795244 (+ 5912us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:52573: BlockingRecv error: recv got EOF from 127.0.98.2:52573 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 29 19:01:34 dist-test-slave-lqkv krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 19:01:34 dist-test-slave-lqkv krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 19:01:34.903944 9590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:34.899419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35045 (local address 127.0.98.3:49313)
0929 19:01:34.899594 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:34.899605 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 19:01:34.899698 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:01:34.899849 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:01:34.899850 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:01:34.900029 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:01:34.900074 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:34.900076 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:34.900724 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:34.900725 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:34.901283 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:34.901285 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:34.901641 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:34.901642 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:34.901697 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:34.902113 (+ 416us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:01:34.902117 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:01:34.903849 (+ 1732us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35045: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":104,"threads_started":1}
W20260929 19:01:34.903995 9588 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:01:34.899495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49313 (local address 127.0.98.62:35045)
0929 19:01:34.899577 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:34.899579 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:34.899679 (+ 100us) server_negotiation.cc:407] Connection header received
0929 19:01:34.899755 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:34.899756 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:34.899771 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:34.899858 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:01:34.900155 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:34.900638 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:34.901380 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:34.901559 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:34.901641 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:34.903929 (+ 2288us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:49313: BlockingRecv error: recv got EOF from 127.0.98.3:49313 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 29 19:01:35 dist-test-slave-lqkv krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 19:01:35 dist-test-slave-lqkv krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 19:01:35.669811 9591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:35.663747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35045 (local address 127.0.98.1:34723)
0929 19:01:35.663877 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:35.663889 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 19:01:35.663952 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:01:35.664087 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:01:35.664088 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:01:35.664354 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:01:35.664429 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:35.664432 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:35.665476 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:35.665477 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:35.666357 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:35.666360 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:35.666935 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:35.666936 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:35.667000 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:35.667578 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:01:35.667584 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:01:35.669742 (+ 2158us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35045: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":70,"threads_started":1}
W20260929 19:01:35.669978 9592 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:01:35.663815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34723 (local address 127.0.98.62:35045)
0929 19:01:35.663949 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:35.663951 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:35.663961 (+ 10us) server_negotiation.cc:407] Connection header received
0929 19:01:35.663997 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:35.663998 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:35.664011 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:35.664082 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:01:35.664528 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:35.665400 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:35.666428 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:35.666809 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:35.666876 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:35.669825 (+ 2949us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:34723: BlockingRecv error: recv got EOF from 127.0.98.1:34723 (error 108)
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":78,"threads_started":1}
Sep 29 19:01:35 dist-test-slave-lqkv krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 19:01:35 dist-test-slave-lqkv krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 19:01:35.800339 9593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:35.795719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35045 (local address 127.0.98.2:40019)
0929 19:01:35.795847 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:35.795859 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 19:01:35.795927 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:01:35.796114 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:01:35.796115 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:01:35.796296 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:01:35.796352 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:35.796354 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:35.797056 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:35.797057 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:35.797611 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:35.797612 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:35.797988 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:35.797990 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:35.798047 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:35.798469 (+ 422us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:01:35.798473 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:01:35.800238 (+ 1765us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35045: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":70,"threads_started":1}
W20260929 19:01:35.800379 9592 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:01:35.795817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40019 (local address 127.0.98.62:35045)
0929 19:01:35.795882 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:35.795884 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:35.795908 (+ 24us) server_negotiation.cc:407] Connection header received
0929 19:01:35.796000 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:35.796000 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:35.796012 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:35.796059 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:01:35.796430 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:35.796972 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:35.797692 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:35.797904 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:35.797988 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:35.800320 (+ 2332us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:40019: BlockingRecv error: recv got EOF from 127.0.98.2:40019 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 29 19:01:35 dist-test-slave-lqkv krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 19:01:35 dist-test-slave-lqkv krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 19:01:35.909196 9594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:35.904391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35045 (local address 127.0.98.3:53607)
0929 19:01:35.904569 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:35.904582 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 19:01:35.904681 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:01:35.904880 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:01:35.904882 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:01:35.905064 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:01:35.905109 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:35.905111 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:35.905790 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:35.905791 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:35.906316 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:35.906317 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:35.906716 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:35.906716 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:35.906777 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:35.907206 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:01:35.907212 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:01:35.909106 (+ 1894us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35045: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":95,"threads_started":1}
W20260929 19:01:35.909236 9592 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:01:35.904512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53607 (local address 127.0.98.62:35045)
0929 19:01:35.904572 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:35.904575 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:35.904648 (+ 73us) server_negotiation.cc:407] Connection header received
0929 19:01:35.904740 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:35.904741 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:35.904753 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:35.904856 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:01:35.905194 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:35.905700 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:35.906408 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:35.906599 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:35.906702 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:35.909177 (+ 2475us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:53607: BlockingRecv error: recv got EOF from 127.0.98.3:53607 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Sep 29 19:01:36 dist-test-slave-lqkv krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 19:01:36 dist-test-slave-lqkv krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 19:01:36.674969 9595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:36.670215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35045 (local address 127.0.98.1:51463)
0929 19:01:36.670347 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:36.670360 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 19:01:36.670426 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:01:36.670660 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:01:36.670661 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:01:36.671073 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:01:36.671155 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:36.671158 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:36.671954 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:36.671955 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:36.672470 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:36.672471 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:36.672782 (+ 311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:36.672782 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:36.672846 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:36.673259 (+ 413us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:01:36.673264 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:01:36.674915 (+ 1651us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35045: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":65,"threads_started":1}
W20260929 19:01:36.675077 9596 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:01:36.670296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51463 (local address 127.0.98.62:35045)
0929 19:01:36.670428 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:36.670430 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:36.670438 (+ 8us) server_negotiation.cc:407] Connection header received
0929 19:01:36.670500 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:36.670500 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:36.670513 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:36.670599 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:01:36.671368 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:36.671884 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:36.672540 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:36.672719 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:36.672769 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:36.675004 (+ 2235us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:51463: BlockingRecv error: recv got EOF from 127.0.98.1:51463 (error 108)
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":63,"threads_started":1}
Sep 29 19:01:36 dist-test-slave-lqkv krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 19:01:36 dist-test-slave-lqkv krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 19:01:36.805209 9597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:36.800855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35045 (local address 127.0.98.2:35927)
0929 19:01:36.800990 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:36.801017 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0929 19:01:36.801075 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:01:36.801196 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:01:36.801197 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:01:36.801357 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:01:36.801402 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:36.801404 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:36.802022 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:36.802023 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:36.802538 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:36.802540 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:36.802834 (+ 294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:36.802835 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:36.802869 (+ 34us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:36.803347 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:01:36.803352 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:01:36.805155 (+ 1803us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35045: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":87,"threads_started":1}
W20260929 19:01:36.805330 9596 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:01:36.800870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35927 (local address 127.0.98.62:35045)
0929 19:01:36.800947 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:36.800949 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:36.801077 (+ 128us) server_negotiation.cc:407] Connection header received
0929 19:01:36.801119 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:36.801120 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:36.801129 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:36.801177 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:01:36.801471 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:36.801949 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:36.802603 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:36.802776 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:36.802825 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:36.805236 (+ 2411us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:35927: BlockingRecv error: recv got EOF from 127.0.98.2:35927 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 29 19:01:36 dist-test-slave-lqkv krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 19:01:36 dist-test-slave-lqkv krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 19:01:36.913993 9598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:36.909645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35045 (local address 127.0.98.3:39191)
0929 19:01:36.909790 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:36.909803 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 19:01:36.909869 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:01:36.910051 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:01:36.910052 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:01:36.910220 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:01:36.910264 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:36.910266 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:36.910924 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:36.910926 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:36.911452 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:36.911453 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:36.911786 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:36.911787 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:36.911843 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:36.912262 (+ 419us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:01:36.912266 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:01:36.913898 (+ 1632us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35045: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":53,"threads_started":1}
W20260929 19:01:36.914044 9596 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:01:36.909814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39191 (local address 127.0.98.62:35045)
0929 19:01:36.909884 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:36.909886 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:36.909894 (+ 8us) server_negotiation.cc:407] Connection header received
0929 19:01:36.909946 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:36.909947 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:36.909969 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:36.910043 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:01:36.910343 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:36.910841 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:36.911535 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:36.911716 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:36.911788 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:36.913978 (+ 2190us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:39191: BlockingRecv error: recv got EOF from 127.0.98.3:39191 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 29 19:01:37 dist-test-slave-lqkv krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 19:01:37 dist-test-slave-lqkv krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 19:01:37.679874 9599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:37.675529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35045 (local address 127.0.98.1:36299)
0929 19:01:37.675661 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:37.675673 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 19:01:37.675735 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:01:37.675891 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:01:37.675892 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:01:37.676092 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:01:37.676140 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:37.676142 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:37.676737 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:37.676737 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:37.677286 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:37.677288 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:37.677598 (+ 310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:37.677599 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:37.677646 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:37.678086 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:01:37.678091 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:01:37.679818 (+ 1727us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35045: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":63,"threads_started":1}
W20260929 19:01:37.680006 9600 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:01:37.675586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36299 (local address 127.0.98.62:35045)
0929 19:01:37.675676 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:37.675678 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:37.675716 (+ 38us) server_negotiation.cc:407] Connection header received
0929 19:01:37.675798 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:37.675799 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:37.675813 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:37.675874 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:01:37.676200 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:37.676674 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:37.677345 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:37.677514 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:37.677568 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:37.679900 (+ 2332us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:36299: BlockingRecv error: recv got EOF from 127.0.98.1:36299 (error 108)
Metrics: {"server-negotiator.queue_time_us":68,"thread_start_us":46,"threads_started":1}
Sep 29 19:01:37 dist-test-slave-lqkv krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 19:01:37 dist-test-slave-lqkv krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 19:01:37.810158 9601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:37.805620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35045 (local address 127.0.98.2:52195)
0929 19:01:37.805812 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:37.805824 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 19:01:37.805898 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:01:37.806052 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:01:37.806053 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:01:37.806254 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:01:37.806299 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:37.806301 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:37.806947 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:37.806948 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:37.807500 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:37.807502 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:37.807912 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:37.807912 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:37.807970 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:37.808439 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:01:37.808443 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:01:37.810064 (+ 1621us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35045: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":98,"threads_started":1}
W20260929 19:01:37.810205 9600 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:01:37.805711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52195 (local address 127.0.98.62:35045)
0929 19:01:37.805795 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:37.805797 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:37.805893 (+ 96us) server_negotiation.cc:407] Connection header received
0929 19:01:37.805963 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:37.805964 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:37.805981 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:37.806043 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:01:37.806380 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:37.806866 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:37.807582 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:37.807839 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:37.807914 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:37.810143 (+ 2229us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:52195: BlockingRecv error: recv got EOF from 127.0.98.2:52195 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 29 19:01:37 dist-test-slave-lqkv krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 19:01:37 dist-test-slave-lqkv krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 19:01:37.918776 9602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:37.914545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35045 (local address 127.0.98.3:34389)
0929 19:01:37.914689 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:37.914700 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 19:01:37.914757 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:01:37.914864 (+ 107us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:01:37.914864 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:01:37.915016 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:01:37.915063 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:37.915065 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:37.915656 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:37.915657 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:37.916197 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:37.916199 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:37.916481 (+ 282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:37.916482 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:37.916518 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:37.916951 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:01:37.916955 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:01:37.918672 (+ 1717us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35045: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":63,"threads_started":1}
W20260929 19:01:37.918810 9600 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:01:37.914629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34389 (local address 127.0.98.62:35045)
0929 19:01:37.914697 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:37.914699 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:37.914742 (+ 43us) server_negotiation.cc:407] Connection header received
0929 19:01:37.914808 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:37.914808 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:37.914819 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:37.914858 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:01:37.915117 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:37.915593 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:37.916258 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:37.916421 (+ 163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:37.916468 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:37.918746 (+ 2278us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:34389: BlockingRecv error: recv got EOF from 127.0.98.3:34389 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 29 19:01:38 dist-test-slave-lqkv krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 19:01:38 dist-test-slave-lqkv krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 19:01:38.684541 9603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:38.680387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35045 (local address 127.0.98.1:41821)
0929 19:01:38.680496 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:38.680513 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 19:01:38.680583 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:01:38.680707 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:01:38.680708 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:01:38.680924 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:01:38.680975 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:38.680976 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:38.681596 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:38.681596 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:38.682111 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:38.682112 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:38.682437 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:38.682438 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:38.682474 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:38.682861 (+ 387us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:01:38.682865 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:01:38.684479 (+ 1614us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35045: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":59,"threads_started":1}
W20260929 19:01:38.684628 9604 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:01:38.680431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41821 (local address 127.0.98.62:35045)
0929 19:01:38.680567 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:38.680569 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:38.680578 (+ 9us) server_negotiation.cc:407] Connection header received
0929 19:01:38.680645 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:38.680645 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:38.680654 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:38.680702 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:01:38.681041 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:38.681531 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:38.682190 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:38.682369 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:38.682426 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:38.684561 (+ 2135us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:41821: BlockingRecv error: recv got EOF from 127.0.98.1:41821 (error 108)
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":75,"threads_started":1}
Sep 29 19:01:38 dist-test-slave-lqkv krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 19:01:38 dist-test-slave-lqkv krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 19:01:38.814944 9605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:38.810633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35045 (local address 127.0.98.2:49785)
0929 19:01:38.810772 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:38.810785 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 19:01:38.810847 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:01:38.810965 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:01:38.810966 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:01:38.811142 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:01:38.811187 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:38.811188 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:38.811846 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:38.811847 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:38.812352 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:38.812354 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:38.812698 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:38.812698 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:38.812734 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:38.813213 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:01:38.813217 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:01:38.814887 (+ 1670us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35045: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":93,"threads_started":1}
W20260929 19:01:38.815047 9604 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:01:38.810733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49785 (local address 127.0.98.62:35045)
0929 19:01:38.810799 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:38.810801 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:38.810836 (+ 35us) server_negotiation.cc:407] Connection header received
0929 19:01:38.810892 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:38.810892 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:38.810902 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:38.810950 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:01:38.811261 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:38.811770 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:38.812442 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:38.812631 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:38.812689 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:38.814976 (+ 2287us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:49785: BlockingRecv error: recv got EOF from 127.0.98.2:49785 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Sep 29 19:01:38 dist-test-slave-lqkv krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 19:01:38 dist-test-slave-lqkv krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 19:01:38.923331 9606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:38.919216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35045 (local address 127.0.98.3:38627)
0929 19:01:38.919320 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:38.919337 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 19:01:38.919404 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:01:38.919522 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:01:38.919522 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:01:38.919689 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:01:38.919736 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:38.919737 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:38.920356 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:38.920357 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:38.920931 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:38.920933 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:38.921231 (+ 298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:38.921232 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:38.921267 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:38.921689 (+ 422us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:01:38.921693 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:01:38.923276 (+ 1583us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35045: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":84,"thread_start_us":51,"threads_started":1}
W20260929 19:01:38.923445 9604 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:01:38.919280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38627 (local address 127.0.98.62:35045)
0929 19:01:38.919358 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:38.919359 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:38.919400 (+ 41us) server_negotiation.cc:407] Connection header received
0929 19:01:38.919445 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:38.919445 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:38.919456 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:38.919501 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:01:38.919815 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:38.920293 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:38.920999 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:38.921169 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:38.921220 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:38.923360 (+ 2140us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:38627: BlockingRecv error: recv got EOF from 127.0.98.3:38627 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 29 19:01:39 dist-test-slave-lqkv krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 19:01:39 dist-test-slave-lqkv krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 19:01:39.689443 9607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:39.684982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35045 (local address 127.0.98.1:51355)
0929 19:01:39.685098 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:39.685111 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 19:01:39.685154 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:01:39.685269 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:01:39.685269 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:01:39.685429 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:01:39.685473 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:39.685475 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:39.686086 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:39.686087 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:39.686675 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:39.686677 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:39.687159 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:39.687161 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:39.687219 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:39.687687 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:01:39.687696 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:01:39.689355 (+ 1659us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35045: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":67,"threads_started":1}
W20260929 19:01:39.689495 9608 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:01:39.685041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51355 (local address 127.0.98.62:35045)
0929 19:01:39.685158 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:39.685159 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:39.685168 (+ 9us) server_negotiation.cc:407] Connection header received
0929 19:01:39.685187 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:39.685188 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:39.685197 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:39.685256 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:01:39.685530 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:39.686020 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:39.686753 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:39.687078 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:39.687147 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:39.689429 (+ 2282us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:51355: BlockingRecv error: recv got EOF from 127.0.98.1:51355 (error 108)
Metrics: {"server-negotiator.queue_time_us":92,"thread_start_us":64,"threads_started":1}
Sep 29 19:01:39 dist-test-slave-lqkv krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 19:01:39 dist-test-slave-lqkv krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 19:01:39.823820 9609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:39.815437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35045 (local address 127.0.98.2:47715)
0929 19:01:39.815567 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:39.815583 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 19:01:39.815633 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:01:39.815728 (+ 95us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:01:39.815728 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:01:39.815889 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:01:39.815935 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:39.815937 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:39.816555 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:39.816556 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:39.817104 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:39.817106 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:39.817419 (+ 313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:39.817419 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:39.817455 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:39.822028 (+ 4573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:01:39.822032 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:01:39.823726 (+ 1694us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35045: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":83,"threads_started":1}
W20260929 19:01:39.823858 9608 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:01:39.815491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47715 (local address 127.0.98.62:35045)
0929 19:01:39.815546 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:39.815548 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:39.815642 (+ 94us) server_negotiation.cc:407] Connection header received
0929 19:01:39.815669 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:39.815669 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:39.815679 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:39.815724 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:01:39.815987 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:39.816475 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:39.817168 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:39.817353 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:39.817408 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:39.823800 (+ 6392us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:47715: BlockingRecv error: recv got EOF from 127.0.98.2:47715 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Sep 29 19:01:39 dist-test-slave-lqkv krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 19:01:39 dist-test-slave-lqkv krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 19:01:39.927711 9610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:39.923754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35045 (local address 127.0.98.3:40873)
0929 19:01:39.923876 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:39.923886 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 19:01:39.923939 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:01:39.924055 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:01:39.924056 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:01:39.924203 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:01:39.924249 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:39.924250 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:39.924891 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:39.924892 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:39.925459 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:39.925462 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:39.925758 (+ 296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:39.925759 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:39.925794 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:39.926194 (+ 400us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:01:39.926198 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:01:39.927651 (+ 1453us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35045: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":71,"threads_started":1}
W20260929 19:01:39.927766 9608 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:01:39.923822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40873 (local address 127.0.98.62:35045)
0929 19:01:39.923880 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:39.923882 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:39.923936 (+ 54us) server_negotiation.cc:407] Connection header received
0929 19:01:39.923989 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:39.923990 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:39.924001 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:39.924040 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:01:39.924305 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:39.924793 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:39.925527 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:39.925699 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:39.925748 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:39.927723 (+ 1975us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:40873: BlockingRecv error: recv got EOF from 127.0.98.3:40873 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Sep 29 19:01:40 dist-test-slave-lqkv krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 19:01:40 dist-test-slave-lqkv krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 19:01:40.694537 9611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:40.689954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35045 (local address 127.0.98.1:39057)
0929 19:01:40.690102 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:40.690113 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 19:01:40.690159 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:01:40.690299 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:01:40.690300 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:01:40.690445 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:01:40.690490 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:40.690491 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:40.691119 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:40.691119 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:40.691640 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:40.691641 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:40.692088 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:40.692089 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:40.692149 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:40.692667 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:01:40.692672 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:01:40.694457 (+ 1785us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35045: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":89,"threads_started":1}
W20260929 19:01:40.694670 9612 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:01:40.690046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39057 (local address 127.0.98.62:35045)
0929 19:01:40.690156 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:40.690158 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:40.690165 (+ 7us) server_negotiation.cc:407] Connection header received
0929 19:01:40.690207 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:40.690207 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:40.690217 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:40.690281 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:01:40.690551 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:40.691055 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:40.691752 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:40.692003 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:40.692062 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:40.694557 (+ 2495us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:39057: BlockingRecv error: recv got EOF from 127.0.98.1:39057 (error 108)
Metrics: {"server-negotiator.queue_time_us":94,"thread_start_us":46,"threads_started":1}
Sep 29 19:01:40 dist-test-slave-lqkv krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 29 19:01:40 dist-test-slave-lqkv krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260929 19:01:40.828496 9613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:40.824342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35045 (local address 127.0.98.2:45665)
0929 19:01:40.824485 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:40.824495 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 19:01:40.824548 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:01:40.824666 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:01:40.824667 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:01:40.824844 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:01:40.824888 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:40.824890 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:40.825474 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:40.825474 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:40.825992 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:40.825994 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:40.826293 (+ 299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:40.826294 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:40.826329 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:40.826822 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:01:40.826827 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:01:40.828444 (+ 1617us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:35045: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":85,"threads_started":1}
W20260929 19:01:40.828622 9612 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:01:40.824405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45665 (local address 127.0.98.62:35045)
0929 19:01:40.824473 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:40.824475 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:40.824553 (+ 78us) server_negotiation.cc:407] Connection header received
0929 19:01:40.824590 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:40.824590 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:40.824600 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:40.824646 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:01:40.824935 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:40.825413 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:40.826058 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:40.826232 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:40.826282 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:40.828543 (+ 2261us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:45665: BlockingRecv error: recv got EOF from 127.0.98.2:45665 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 19:01:40.851752 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUbnLHE/build/release/bin/kudu with pid 9135
I20260929 19:01:40.856192 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUbnLHE/build/release/bin/kudu with pid 9286
I20260929 19:01:40.860397 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUbnLHE/build/release/bin/kudu with pid 9419
I20260929 19:01:40.864319 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUbnLHE/build/release/bin/kudu with pid 9064
2026-09-29T19:01:40Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17366 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-29T19:01:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T19:01:40Z Disabled control of system clock
I20260929 19:01:40.890379 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUbnLHE/build/release/bin/kudu
/tmp/dist-test-taskUbnLHE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708447806542-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708447806542-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708447806542-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708447806542-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41397
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:44183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41397
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260929 19:01:40.968403 9617 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:01:40.968580 9617 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:01:40.968600 9617 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:01:40.969745 9617 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 19:01:40.969782 9617 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:01:40.969794 9617 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:01:40.969807 9617 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 19:01:40.969818 9617 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 19:01:40.971021 9617 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708447806542-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708447806542-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41397
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41397
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708447806542-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9617
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708447806542-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 18:09:13 UTC on ccycloud.quasar-hpkalf.root.comops.site
build id 4680913
I20260929 19:01:40.971184 9617 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:01:40.971380 9617 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 19:01:40.976269 9617 server_base.cc:1034] running on GCE node
W20260929 19:01:40.976342 9625 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:01:40.976341 9622 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:01:40.976377 9623 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:01:40.976760 9617 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:01:40.977098 9617 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:01:40.978255 9617 hybrid_clock.cc:648] HybridClock initialized: now 1790708500978234 us; error 32 us; skew 500 ppm
I20260929 19:01:40.979369 9617 webserver.cc:460] Webserver started at http://127.0.98.62:42141/ using document root <none> and password file <none>
I20260929 19:01:40.979588 9617 fs_manager.cc:359] Metadata directory not provided
I20260929 19:01:40.979653 9617 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:01:40.979807 9617 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:01:40.980785 9617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708447806542-392-0/minicluster-data/master-0/data/instance:
uuid: "3da9e7efd8a449ee9c9263fd7e8188d4"
format_stamp: "Formatted at 2026-09-29 19:01:40 on dist-test-slave-lqkv"
server_key: "2f3ec26f0620db86bf5a7268cbf4cc7a"
server_key_iv: "3c32ef657775756f353fa277f237674d"
server_key_version: "encryptionkey@0"
I20260929 19:01:40.981204 9617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708447806542-392-0/minicluster-data/master-0/wal/instance:
uuid: "3da9e7efd8a449ee9c9263fd7e8188d4"
format_stamp: "Formatted at 2026-09-29 19:01:40 on dist-test-slave-lqkv"
server_key: "2f3ec26f0620db86bf5a7268cbf4cc7a"
server_key_iv: "3c32ef657775756f353fa277f237674d"
server_key_version: "encryptionkey@0"
I20260929 19:01:40.982395 9617 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 19:01:40.983099 9631 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:40.983273 9617 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:40.983363 9617 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708447806542-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708447806542-392-0/minicluster-data/master-0/wal
uuid: "3da9e7efd8a449ee9c9263fd7e8188d4"
format_stamp: "Formatted at 2026-09-29 19:01:40 on dist-test-slave-lqkv"
server_key: "2f3ec26f0620db86bf5a7268cbf4cc7a"
server_key_iv: "3c32ef657775756f353fa277f237674d"
server_key_version: "encryptionkey@0"
I20260929 19:01:40.983448 9617 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708447806542-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708447806542-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708447806542-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:01:40.997004 9617 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:01:40.997248 9617 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:01:40.997391 9617 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:01:41.001148 9617 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41397
I20260929 19:01:41.001176 9683 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41397 every 8 connection(s)
I20260929 19:01:41.001474 9617 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708447806542-392-0/minicluster-data/master-0/data/info.pb
I20260929 19:01:41.002063 9684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:01:41.004024 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUbnLHE/build/release/bin/kudu as pid 9617
I20260929 19:01:41.004136 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708447806542-392-0/minicluster-data/master-0/wal/instance
I20260929 19:01:41.004308 392 external_mini_cluster.cc:1428] Setting key 0514e8452c0af1ac95705842e1dee650
I20260929 19:01:41.004534 9684 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3da9e7efd8a449ee9c9263fd7e8188d4: Bootstrap starting.
I20260929 19:01:41.005251 9684 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3da9e7efd8a449ee9c9263fd7e8188d4: Neither blocks nor log segments found. Creating new log.
I20260929 19:01:41.005558 9684 log.cc:824] T 00000000000000000000000000000000 P 3da9e7efd8a449ee9c9263fd7e8188d4: Log is configured to *not* fsync() on all Append() calls
I20260929 19:01:41.006296 9684 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3da9e7efd8a449ee9c9263fd7e8188d4: No bootstrap required, opened a new log
I20260929 19:01:41.008395 9684 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3da9e7efd8a449ee9c9263fd7e8188d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3da9e7efd8a449ee9c9263fd7e8188d4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41397 } }
I20260929 19:01:41.008546 9684 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3da9e7efd8a449ee9c9263fd7e8188d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:01:41.008579 9684 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3da9e7efd8a449ee9c9263fd7e8188d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3da9e7efd8a449ee9c9263fd7e8188d4, State: Initialized, Role: FOLLOWER
I20260929 19:01:41.008685 9684 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3da9e7efd8a449ee9c9263fd7e8188d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3da9e7efd8a449ee9c9263fd7e8188d4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41397 } }
I20260929 19:01:41.008772 9684 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3da9e7efd8a449ee9c9263fd7e8188d4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 19:01:41.008833 9684 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3da9e7efd8a449ee9c9263fd7e8188d4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 19:01:41.008925 9684 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3da9e7efd8a449ee9c9263fd7e8188d4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:01:41.009611 9684 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3da9e7efd8a449ee9c9263fd7e8188d4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3da9e7efd8a449ee9c9263fd7e8188d4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41397 } }
I20260929 19:01:41.009721 9684 leader_election.cc:302] T 00000000000000000000000000000000 P 3da9e7efd8a449ee9c9263fd7e8188d4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3da9e7efd8a449ee9c9263fd7e8188d4; no voters:
I20260929 19:01:41.009868 9684 leader_election.cc:288] T 00000000000000000000000000000000 P 3da9e7efd8a449ee9c9263fd7e8188d4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 19:01:41.009905 9689 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3da9e7efd8a449ee9c9263fd7e8188d4 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:01:41.010056 9689 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3da9e7efd8a449ee9c9263fd7e8188d4 [term 1 LEADER]: Becoming Leader. State: Replica: 3da9e7efd8a449ee9c9263fd7e8188d4, State: Running, Role: LEADER
I20260929 19:01:41.010147 9684 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3da9e7efd8a449ee9c9263fd7e8188d4 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 19:01:41.010190 9689 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3da9e7efd8a449ee9c9263fd7e8188d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3da9e7efd8a449ee9c9263fd7e8188d4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41397 } }
I20260929 19:01:41.010372 9687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:41.004799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47994 (local address 127.0.98.62:41397)
0929 19:01:41.005053 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:41.005061 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:41.005092 (+ 31us) server_negotiation.cc:407] Connection header received
0929 19:01:41.005686 (+ 594us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:41.005693 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:41.005771 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:41.005890 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:01:41.006127 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:41.007422 (+ 1295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:41.008172 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:41.008766 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:41.008929 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:41.009566 (+ 637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:01:41.009570 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:01:41.009570 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 19:01:41.009762 (+ 192us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:01:41.009803 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 19:01:41.009818 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:01:41.009880 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0929 19:01:41.010088 (+ 208us) server_negotiation.cc:299] Negotiation successful
0929 19:01:41.010178 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":107,"threads_started":1}
I20260929 19:01:41.010771 9690 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3da9e7efd8a449ee9c9263fd7e8188d4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3da9e7efd8a449ee9c9263fd7e8188d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3da9e7efd8a449ee9c9263fd7e8188d4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41397 } } }
I20260929 19:01:41.010932 9690 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3da9e7efd8a449ee9c9263fd7e8188d4 [sys.catalog]: This master's current role is: LEADER
I20260929 19:01:41.011053 9691 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3da9e7efd8a449ee9c9263fd7e8188d4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3da9e7efd8a449ee9c9263fd7e8188d4. Latest consensus state: current_term: 1 leader_uuid: "3da9e7efd8a449ee9c9263fd7e8188d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3da9e7efd8a449ee9c9263fd7e8188d4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41397 } } }
I20260929 19:01:41.011162 9699 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 19:01:41.011193 9691 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3da9e7efd8a449ee9c9263fd7e8188d4 [sys.catalog]: This master's current role is: LEADER
I20260929 19:01:41.011655 9699 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 19:01:41.012995 9699 catalog_manager.cc:1347] Generated new cluster ID: a1da89090e314ec994ac8c35d1dbbec8
I20260929 19:01:41.013063 9699 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 19:01:41.016780 9699 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 19:01:41.017397 9699 catalog_manager.cc:1504] Loading token signing keys...
I20260929 19:01:41.023705 9699 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3da9e7efd8a449ee9c9263fd7e8188d4: Generated new TSK 0
I20260929 19:01:41.023865 9699 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 19:01:41.030019 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUbnLHE/build/release/bin/kudu
/tmp/dist-test-taskUbnLHE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708447806542-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708447806542-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708447806542-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708447806542-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41397
--builtin_ntp_servers=127.0.98.20:44183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260929 19:01:41.118721 9708 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:01:41.118889 9708 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:01:41.118908 9708 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:01:41.120012 9708 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:01:41.120050 9708 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:01:41.120074 9708 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 19:01:41.121461 9708 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708447806542-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708447806542-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708447806542-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41397
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9708
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708447806542-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 18:09:13 UTC on ccycloud.quasar-hpkalf.root.comops.site
build id 4680913
I20260929 19:01:41.121685 9708 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:01:41.121971 9708 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:01:41.127533 9713 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:01:41.127504 9714 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:01:41.127514 9716 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:01:41.127678 9708 server_base.cc:1034] running on GCE node
I20260929 19:01:41.127853 9708 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:01:41.128104 9708 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:01:41.129274 9708 hybrid_clock.cc:648] HybridClock initialized: now 1790708501129261 us; error 25 us; skew 500 ppm
I20260929 19:01:41.130558 9708 webserver.cc:460] Webserver started at http://127.0.98.1:35051/ using document root <none> and password file <none>
I20260929 19:01:41.130808 9708 fs_manager.cc:359] Metadata directory not provided
I20260929 19:01:41.130874 9708 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:01:41.131029 9708 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:01:41.132345 9708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708447806542-392-0/minicluster-data/ts-0/data/instance:
uuid: "0ddfc61633b8471185ba65c3fda8755b"
format_stamp: "Formatted at 2026-09-29 19:01:41 on dist-test-slave-lqkv"
server_key: "8734242864132126933120e5d07502d4"
server_key_iv: "a210f2ac62bf468bd7aa3379355f60e3"
server_key_version: "encryptionkey@0"
I20260929 19:01:41.132740 9708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708447806542-392-0/minicluster-data/ts-0/wal/instance:
uuid: "0ddfc61633b8471185ba65c3fda8755b"
format_stamp: "Formatted at 2026-09-29 19:01:41 on dist-test-slave-lqkv"
server_key: "8734242864132126933120e5d07502d4"
server_key_iv: "a210f2ac62bf468bd7aa3379355f60e3"
server_key_version: "encryptionkey@0"
I20260929 19:01:41.134348 9708 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 19:01:41.135121 9722 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:41.135299 9708 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 19:01:41.135378 9708 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708447806542-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708447806542-392-0/minicluster-data/ts-0/wal
uuid: "0ddfc61633b8471185ba65c3fda8755b"
format_stamp: "Formatted at 2026-09-29 19:01:41 on dist-test-slave-lqkv"
server_key: "8734242864132126933120e5d07502d4"
server_key_iv: "a210f2ac62bf468bd7aa3379355f60e3"
server_key_version: "encryptionkey@0"
I20260929 19:01:41.135439 9708 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708447806542-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708447806542-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708447806542-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:01:41.150372 9708 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:01:41.150650 9708 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:01:41.150782 9708 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:01:41.151011 9708 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:01:41.151355 9708 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:01:41.151407 9708 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:41.151425 9708 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:01:41.151463 9708 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:41.157855 9708 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37967
I20260929 19:01:41.157913 9835 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37967 every 8 connection(s)
I20260929 19:01:41.158182 9708 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708447806542-392-0/minicluster-data/ts-0/data/info.pb
I20260929 19:01:41.163013 9687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:41.158852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41263 (local address 127.0.98.62:41397)
0929 19:01:41.158911 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:41.158913 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:41.159094 (+ 181us) server_negotiation.cc:407] Connection header received
0929 19:01:41.159566 (+ 472us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:41.159567 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:41.159580 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:41.159643 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:01:41.160261 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:41.160738 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:41.161732 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:41.161935 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:41.162009 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:41.162756 (+ 747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:01:41.162757 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:01:41.162757 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 19:01:41.162762 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:01:41.162775 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 19:01:41.162779 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:01:41.162826 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0929 19:01:41.162914 (+ 88us) server_negotiation.cc:299] Negotiation successful
0929 19:01:41.162938 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260929 19:01:41.163235 9838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:41.158809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41397 (local address 127.0.98.1:41263)
0929 19:01:41.158972 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:41.159013 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 19:01:41.159468 (+ 455us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:01:41.159708 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:01:41.159711 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:01:41.159718 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:01:41.160175 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:41.160178 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:41.160846 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:41.160848 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:41.161632 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:41.161634 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:41.162017 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:41.162018 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:41.162159 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:41.162623 (+ 464us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 19:01:41.162623 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:01:41.162648 (+ 25us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 19:01:41.162668 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:01:41.162825 (+ 157us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:01:41.162829 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:01:41.162832 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 19:01:41.162916 (+ 84us) client_negotiation.cc:240] Negotiation successful
0929 19:01:41.163019 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":61,"threads_started":1}
I20260929 19:01:41.163690 9836 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41397
I20260929 19:01:41.163805 9836 heartbeater.cc:464] Registering TS with master...
I20260929 19:01:41.163973 9836 heartbeater.cc:511] Master 127.0.98.62:41397 requested a full tablet report, sending...
I20260929 19:01:41.164435 9648 ts_manager.cc:194] Registered new tserver with Master: 0ddfc61633b8471185ba65c3fda8755b (127.0.98.1:37967)
I20260929 19:01:41.165060 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUbnLHE/build/release/bin/kudu as pid 9708
I20260929 19:01:41.165133 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708447806542-392-0/minicluster-data/ts-0/wal/instance
I20260929 19:01:41.165218 9648 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:41263
I20260929 19:01:41.165335 392 external_mini_cluster.cc:1428] Setting key ad1e0e024e390b0cb91b0acffa5f28fe
I20260929 19:01:41.166173 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUbnLHE/build/release/bin/kudu
/tmp/dist-test-taskUbnLHE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708447806542-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708447806542-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708447806542-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708447806542-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41397
--builtin_ntp_servers=127.0.98.20:44183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260929 19:01:41.243016 9839 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:01:41.243196 9839 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:01:41.243216 9839 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:01:41.244355 9839 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:01:41.244395 9839 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:01:41.244442 9839 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 19:01:41.245751 9839 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708447806542-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708447806542-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708447806542-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41397
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9839
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708447806542-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 18:09:13 UTC on ccycloud.quasar-hpkalf.root.comops.site
build id 4680913
I20260929 19:01:41.245996 9839 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:01:41.246281 9839 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:01:41.250885 9847 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:01:41.250880 9845 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:01:41.251047 9839 server_base.cc:1034] running on GCE node
W20260929 19:01:41.250949 9844 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:01:41.251385 9839 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:01:41.251612 9839 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:01:41.252746 9839 hybrid_clock.cc:648] HybridClock initialized: now 1790708501252739 us; error 27 us; skew 500 ppm
I20260929 19:01:41.253904 9839 webserver.cc:460] Webserver started at http://127.0.98.2:44921/ using document root <none> and password file <none>
I20260929 19:01:41.254137 9839 fs_manager.cc:359] Metadata directory not provided
I20260929 19:01:41.254179 9839 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:01:41.254340 9839 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:01:41.255303 9839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708447806542-392-0/minicluster-data/ts-1/data/instance:
uuid: "837bf906247e4c40a0fbfef7c360d6a1"
format_stamp: "Formatted at 2026-09-29 19:01:41 on dist-test-slave-lqkv"
server_key: "5325b5383d4a650fe47bf9f33a4c6251"
server_key_iv: "35c8349bdc5d8c8919535adea10f280b"
server_key_version: "encryptionkey@0"
I20260929 19:01:41.255610 9839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708447806542-392-0/minicluster-data/ts-1/wal/instance:
uuid: "837bf906247e4c40a0fbfef7c360d6a1"
format_stamp: "Formatted at 2026-09-29 19:01:41 on dist-test-slave-lqkv"
server_key: "5325b5383d4a650fe47bf9f33a4c6251"
server_key_iv: "35c8349bdc5d8c8919535adea10f280b"
server_key_version: "encryptionkey@0"
I20260929 19:01:41.256744 9839 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 19:01:41.257423 9853 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:41.257579 9839 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 19:01:41.257671 9839 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708447806542-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708447806542-392-0/minicluster-data/ts-1/wal
uuid: "837bf906247e4c40a0fbfef7c360d6a1"
format_stamp: "Formatted at 2026-09-29 19:01:41 on dist-test-slave-lqkv"
server_key: "5325b5383d4a650fe47bf9f33a4c6251"
server_key_iv: "35c8349bdc5d8c8919535adea10f280b"
server_key_version: "encryptionkey@0"
I20260929 19:01:41.257720 9839 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708447806542-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708447806542-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708447806542-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:01:41.275084 9839 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:01:41.275295 9839 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:01:41.275401 9839 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:01:41.275555 9839 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:01:41.275838 9839 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:01:41.275871 9839 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:41.275925 9839 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:01:41.275961 9839 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:41.281405 9839 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44045
I20260929 19:01:41.281446 9966 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44045 every 8 connection(s)
I20260929 19:01:41.281713 9839 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708447806542-392-0/minicluster-data/ts-1/data/info.pb
I20260929 19:01:41.286402 9687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:41.282298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39897 (local address 127.0.98.62:41397)
0929 19:01:41.282339 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:41.282340 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:41.282583 (+ 243us) server_negotiation.cc:407] Connection header received
0929 19:01:41.283061 (+ 478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:41.283062 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:41.283073 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:41.283121 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:01:41.283738 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:41.284228 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:41.285140 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:41.285352 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:41.285441 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:41.286118 (+ 677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:01:41.286119 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:01:41.286120 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 19:01:41.286124 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:01:41.286137 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 19:01:41.286140 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:01:41.286191 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0929 19:01:41.286333 (+ 142us) server_negotiation.cc:299] Negotiation successful
0929 19:01:41.286351 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260929 19:01:41.286566 9969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:41.282261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41397 (local address 127.0.98.2:39897)
0929 19:01:41.282434 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:41.282496 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0929 19:01:41.282958 (+ 462us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:01:41.283178 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:01:41.283182 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:01:41.283191 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:01:41.283649 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:41.283653 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:41.284318 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:41.284319 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:41.285049 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:41.285051 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:41.285452 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:41.285453 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:41.285571 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:41.285973 (+ 402us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 19:01:41.285973 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:01:41.285996 (+ 23us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 19:01:41.286015 (+ 19us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:01:41.286235 (+ 220us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:01:41.286241 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:01:41.286244 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 19:01:41.286313 (+ 69us) client_negotiation.cc:240] Negotiation successful
0929 19:01:41.286384 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":68,"threads_started":1}
I20260929 19:01:41.287057 9967 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41397
I20260929 19:01:41.287142 9967 heartbeater.cc:464] Registering TS with master...
I20260929 19:01:41.287284 9967 heartbeater.cc:511] Master 127.0.98.62:41397 requested a full tablet report, sending...
I20260929 19:01:41.287599 9648 ts_manager.cc:194] Registered new tserver with Master: 837bf906247e4c40a0fbfef7c360d6a1 (127.0.98.2:44045)
I20260929 19:01:41.287889 9648 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:39897
I20260929 19:01:41.290175 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUbnLHE/build/release/bin/kudu as pid 9839
I20260929 19:01:41.290241 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708447806542-392-0/minicluster-data/ts-1/wal/instance
I20260929 19:01:41.290381 392 external_mini_cluster.cc:1428] Setting key 790f9f1217604f25ce51d3d91066487b
I20260929 19:01:41.291141 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUbnLHE/build/release/bin/kudu
/tmp/dist-test-taskUbnLHE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708447806542-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708447806542-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708447806542-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708447806542-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41397
--builtin_ntp_servers=127.0.98.20:44183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260929 19:01:41.367547 9970 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:01:41.367713 9970 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:01:41.367731 9970 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:01:41.368732 9970 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:01:41.368762 9970 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:01:41.368784 9970 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 19:01:41.370071 9970 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708447806542-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708447806542-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708447806542-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41397
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9970
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708447806542-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 18:09:13 UTC on ccycloud.quasar-hpkalf.root.comops.site
build id 4680913
I20260929 19:01:41.370232 9970 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:01:41.370463 9970 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:01:41.375231 9975 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:01:41.375237 9976 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:01:41.375237 9978 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:01:41.375341 9970 server_base.cc:1034] running on GCE node
I20260929 19:01:41.375541 9970 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:01:41.375787 9970 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:01:41.376930 9970 hybrid_clock.cc:648] HybridClock initialized: now 1790708501376922 us; error 32 us; skew 500 ppm
I20260929 19:01:41.378126 9970 webserver.cc:460] Webserver started at http://127.0.98.3:39605/ using document root <none> and password file <none>
I20260929 19:01:41.378363 9970 fs_manager.cc:359] Metadata directory not provided
I20260929 19:01:41.378445 9970 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:01:41.378575 9970 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:01:41.379542 9970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708447806542-392-0/minicluster-data/ts-2/data/instance:
uuid: "69c31bac6e9847e7b9b40eb0d97fc963"
format_stamp: "Formatted at 2026-09-29 19:01:41 on dist-test-slave-lqkv"
server_key: "2e2c9bde44615fa15f656c86efbed505"
server_key_iv: "5695575d139ca9f26741d350573f97e5"
server_key_version: "encryptionkey@0"
I20260929 19:01:41.379928 9970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708447806542-392-0/minicluster-data/ts-2/wal/instance:
uuid: "69c31bac6e9847e7b9b40eb0d97fc963"
format_stamp: "Formatted at 2026-09-29 19:01:41 on dist-test-slave-lqkv"
server_key: "2e2c9bde44615fa15f656c86efbed505"
server_key_iv: "5695575d139ca9f26741d350573f97e5"
server_key_version: "encryptionkey@0"
I20260929 19:01:41.381199 9970 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 19:01:41.381847 9984 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:41.381991 9970 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 19:01:41.382074 9970 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708447806542-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708447806542-392-0/minicluster-data/ts-2/wal
uuid: "69c31bac6e9847e7b9b40eb0d97fc963"
format_stamp: "Formatted at 2026-09-29 19:01:41 on dist-test-slave-lqkv"
server_key: "2e2c9bde44615fa15f656c86efbed505"
server_key_iv: "5695575d139ca9f26741d350573f97e5"
server_key_version: "encryptionkey@0"
I20260929 19:01:41.382158 9970 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708447806542-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708447806542-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708447806542-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:01:41.394574 9970 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:01:41.394822 9970 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:01:41.394958 9970 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:01:41.395151 9970 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:01:41.395437 9970 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:01:41.395488 9970 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:41.395524 9970 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:01:41.395550 9970 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:41.400651 9970 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37231
I20260929 19:01:41.400714 10097 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37231 every 8 connection(s)
I20260929 19:01:41.401077 9970 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708447806542-392-0/minicluster-data/ts-2/data/info.pb
I20260929 19:01:41.405514 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUbnLHE/build/release/bin/kudu as pid 9970
I20260929 19:01:41.405551 9687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:41.401684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38443 (local address 127.0.98.62:41397)
0929 19:01:41.401747 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:41.401749 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:41.401932 (+ 183us) server_negotiation.cc:407] Connection header received
0929 19:01:41.402374 (+ 442us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:41.402374 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:41.402383 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:41.402435 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:01:41.403027 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:41.403497 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:41.404362 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:41.404530 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:41.404576 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:41.405324 (+ 748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:01:41.405325 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:01:41.405325 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 19:01:41.405328 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:01:41.405338 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 19:01:41.405341 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:01:41.405378 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0929 19:01:41.405484 (+ 106us) server_negotiation.cc:299] Negotiation successful
0929 19:01:41.405501 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260929 19:01:41.405607 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790708447806542-392-0/minicluster-data/ts-2/wal/instance
I20260929 19:01:41.405797 392 external_mini_cluster.cc:1428] Setting key 0406b1f46e4b758b754f46acc594ff2f
I20260929 19:01:41.405792 10100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:41.401634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41397 (local address 127.0.98.3:38443)
0929 19:01:41.401813 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:41.401851 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 19:01:41.402296 (+ 445us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:01:41.402499 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:01:41.402501 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:01:41.402509 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:01:41.402959 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:41.402963 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:41.403568 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:41.403568 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:41.404294 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:41.404295 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:41.404595 (+ 300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:41.404596 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:41.404691 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:41.405200 (+ 509us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 19:01:41.405200 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:01:41.405224 (+ 24us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 19:01:41.405244 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:01:41.405399 (+ 155us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:01:41.405404 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:01:41.405407 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 19:01:41.405485 (+ 78us) client_negotiation.cc:240] Negotiation successful
0929 19:01:41.405587 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":88,"threads_started":1}
I20260929 19:01:41.406246 10098 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41397
I20260929 19:01:41.406324 10098 heartbeater.cc:464] Registering TS with master...
I20260929 19:01:41.406512 10098 heartbeater.cc:511] Master 127.0.98.62:41397 requested a full tablet report, sending...
I20260929 19:01:41.406857 9647 ts_manager.cc:194] Registered new tserver with Master: 69c31bac6e9847e7b9b40eb0d97fc963 (127.0.98.3:37231)
I20260929 19:01:41.407217 9647 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:38443
I20260929 19:01:41.417537 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260929 19:01:41.419193 9687 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:01:41.418740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48006 (local address 127.0.98.62:41397)
0929 19:01:41.418798 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:41.418800 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:41.418896 (+ 96us) server_negotiation.cc:407] Connection header received
0929 19:01:41.418942 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:41.418942 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:41.418949 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:41.418992 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:01:41.419132 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48006: BlockingRecv error: recv got EOF from 127.0.0.1:48006 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
I20260929 19:01:41.419247 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUbnLHE/build/release/bin/kudu with pid 9708
I20260929 19:01:41.423954 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUbnLHE/build/release/bin/kudu with pid 9839
I20260929 19:01:41.428171 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUbnLHE/build/release/bin/kudu with pid 9970
I20260929 19:01:41.431903 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUbnLHE/build/release/bin/kudu with pid 9617
2026-09-29T19:01:41Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (567 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-29T19:01:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T19:01:41Z Disabled control of system clock
I20260929 19:01:41.457653 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUbnLHE/build/release/bin/kudu
/tmp/dist-test-taskUbnLHE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708447806542-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708447806542-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708447806542-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708447806542-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46663
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:33551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46663
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260929 19:01:41.531596 10109 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:01:41.531785 10109 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:01:41.531803 10109 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:01:41.532850 10109 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 19:01:41.532881 10109 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:01:41.532893 10109 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:01:41.532907 10109 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 19:01:41.532917 10109 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 19:01:41.534144 10109 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708447806542-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708447806542-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46663
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46663
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708447806542-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10109
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708447806542-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 18:09:13 UTC on ccycloud.quasar-hpkalf.root.comops.site
build id 4680913
I20260929 19:01:41.534341 10109 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:01:41.534538 10109 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:01:41.538902 10114 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:01:41.538920 10117 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:01:41.539103 10109 server_base.cc:1034] running on GCE node
W20260929 19:01:41.539073 10115 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:01:41.539435 10109 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:01:41.539701 10109 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:01:41.540879 10109 hybrid_clock.cc:648] HybridClock initialized: now 1790708501540853 us; error 45 us; skew 500 ppm
I20260929 19:01:41.542028 10109 webserver.cc:460] Webserver started at http://127.0.98.62:34193/ using document root <none> and password file <none>
I20260929 19:01:41.542249 10109 fs_manager.cc:359] Metadata directory not provided
I20260929 19:01:41.542292 10109 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:01:41.542434 10109 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:01:41.543457 10109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708447806542-392-0/minicluster-data/master-0/data/instance:
uuid: "58e2e6cceb034df5ba601e4f7a2639d0"
format_stamp: "Formatted at 2026-09-29 19:01:41 on dist-test-slave-lqkv"
server_key: "7d6a8280db4edadad83d335afde8fabc"
server_key_iv: "2e68077a474fd40b71e4bd394ebe7860"
server_key_version: "encryptionkey@0"
I20260929 19:01:41.543798 10109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708447806542-392-0/minicluster-data/master-0/wal/instance:
uuid: "58e2e6cceb034df5ba601e4f7a2639d0"
format_stamp: "Formatted at 2026-09-29 19:01:41 on dist-test-slave-lqkv"
server_key: "7d6a8280db4edadad83d335afde8fabc"
server_key_iv: "2e68077a474fd40b71e4bd394ebe7860"
server_key_version: "encryptionkey@0"
I20260929 19:01:41.545032 10109 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 19:01:41.545674 10123 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:41.545832 10109 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 19:01:41.545919 10109 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708447806542-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708447806542-392-0/minicluster-data/master-0/wal
uuid: "58e2e6cceb034df5ba601e4f7a2639d0"
format_stamp: "Formatted at 2026-09-29 19:01:41 on dist-test-slave-lqkv"
server_key: "7d6a8280db4edadad83d335afde8fabc"
server_key_iv: "2e68077a474fd40b71e4bd394ebe7860"
server_key_version: "encryptionkey@0"
I20260929 19:01:41.545993 10109 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708447806542-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708447806542-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708447806542-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:01:41.549084 10109 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:01:41.549324 10109 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:01:41.549455 10109 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:01:41.553145 10109 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46663
I20260929 19:01:41.553177 10175 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46663 every 8 connection(s)
I20260929 19:01:41.553471 10109 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708447806542-392-0/minicluster-data/master-0/data/info.pb
I20260929 19:01:41.554052 10176 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:01:41.556453 10176 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58e2e6cceb034df5ba601e4f7a2639d0: Bootstrap starting.
I20260929 19:01:41.557003 10176 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 58e2e6cceb034df5ba601e4f7a2639d0: Neither blocks nor log segments found. Creating new log.
I20260929 19:01:41.557268 10176 log.cc:824] T 00000000000000000000000000000000 P 58e2e6cceb034df5ba601e4f7a2639d0: Log is configured to *not* fsync() on all Append() calls
I20260929 19:01:41.557838 10176 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58e2e6cceb034df5ba601e4f7a2639d0: No bootstrap required, opened a new log
I20260929 19:01:41.559262 10176 raft_consensus.cc:348] T 00000000000000000000000000000000 P 58e2e6cceb034df5ba601e4f7a2639d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58e2e6cceb034df5ba601e4f7a2639d0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46663 } }
I20260929 19:01:41.559384 10176 raft_consensus.cc:374] T 00000000000000000000000000000000 P 58e2e6cceb034df5ba601e4f7a2639d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:01:41.559405 10176 raft_consensus.cc:729] T 00000000000000000000000000000000 P 58e2e6cceb034df5ba601e4f7a2639d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58e2e6cceb034df5ba601e4f7a2639d0, State: Initialized, Role: FOLLOWER
I20260929 19:01:41.559562 10176 consensus_queue.cc:260] T 00000000000000000000000000000000 P 58e2e6cceb034df5ba601e4f7a2639d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58e2e6cceb034df5ba601e4f7a2639d0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46663 } }
I20260929 19:01:41.559620 10176 raft_consensus.cc:388] T 00000000000000000000000000000000 P 58e2e6cceb034df5ba601e4f7a2639d0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 19:01:41.559640 10176 raft_consensus.cc:482] T 00000000000000000000000000000000 P 58e2e6cceb034df5ba601e4f7a2639d0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 19:01:41.559706 10176 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 58e2e6cceb034df5ba601e4f7a2639d0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:01:41.560173 10176 raft_consensus.cc:504] T 00000000000000000000000000000000 P 58e2e6cceb034df5ba601e4f7a2639d0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58e2e6cceb034df5ba601e4f7a2639d0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46663 } }
I20260929 19:01:41.560284 10176 leader_election.cc:302] T 00000000000000000000000000000000 P 58e2e6cceb034df5ba601e4f7a2639d0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 58e2e6cceb034df5ba601e4f7a2639d0; no voters:
I20260929 19:01:41.560451 10176 leader_election.cc:288] T 00000000000000000000000000000000 P 58e2e6cceb034df5ba601e4f7a2639d0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 19:01:41.560514 10179 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 58e2e6cceb034df5ba601e4f7a2639d0 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:01:41.560660 10179 raft_consensus.cc:686] T 00000000000000000000000000000000 P 58e2e6cceb034df5ba601e4f7a2639d0 [term 1 LEADER]: Becoming Leader. State: Replica: 58e2e6cceb034df5ba601e4f7a2639d0, State: Running, Role: LEADER
I20260929 19:01:41.560674 10176 sys_catalog.cc:564] T 00000000000000000000000000000000 P 58e2e6cceb034df5ba601e4f7a2639d0 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 19:01:41.560799 10179 consensus_queue.cc:237] T 00000000000000000000000000000000 P 58e2e6cceb034df5ba601e4f7a2639d0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58e2e6cceb034df5ba601e4f7a2639d0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46663 } }
I20260929 19:01:41.561300 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUbnLHE/build/release/bin/kudu as pid 10109
I20260929 19:01:41.561404 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708447806542-392-0/minicluster-data/master-0/wal/instance
I20260929 19:01:41.561404 10180 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58e2e6cceb034df5ba601e4f7a2639d0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "58e2e6cceb034df5ba601e4f7a2639d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58e2e6cceb034df5ba601e4f7a2639d0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46663 } } }
I20260929 19:01:41.561481 10180 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58e2e6cceb034df5ba601e4f7a2639d0 [sys.catalog]: This master's current role is: LEADER
I20260929 19:01:41.561604 392 external_mini_cluster.cc:1428] Setting key 5740a8aaf164f0f0f2171970d7c2d096
I20260929 19:01:41.561689 10188 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 19:01:41.561869 10181 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58e2e6cceb034df5ba601e4f7a2639d0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 58e2e6cceb034df5ba601e4f7a2639d0. Latest consensus state: current_term: 1 leader_uuid: "58e2e6cceb034df5ba601e4f7a2639d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58e2e6cceb034df5ba601e4f7a2639d0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46663 } } }
I20260929 19:01:41.561939 10181 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58e2e6cceb034df5ba601e4f7a2639d0 [sys.catalog]: This master's current role is: LEADER
I20260929 19:01:41.562244 10188 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 19:01:41.564379 10196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:41.562185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60388 (local address 127.0.98.62:46663)
0929 19:01:41.562544 (+ 359us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:41.562552 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:41.562582 (+ 30us) server_negotiation.cc:407] Connection header received
0929 19:01:41.563016 (+ 434us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:41.563041 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:41.563089 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:41.563255 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:01:41.563924 (+ 669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:01:41.563928 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:01:41.563928 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 19:01:41.563932 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:01:41.563973 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 19:01:41.563983 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:01:41.564034 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0929 19:01:41.564141 (+ 107us) server_negotiation.cc:299] Negotiation successful
0929 19:01:41.564214 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":67,"threads_started":1}
I20260929 19:01:41.568625 10188 catalog_manager.cc:1347] Generated new cluster ID: f4509f8f33f54680a4a4b61817e866d4
I20260929 19:01:41.568676 10188 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 19:01:41.585963 10188 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 19:01:41.586117 10188 catalog_manager.cc:1504] Loading token signing keys...
I20260929 19:01:41.594365 10188 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 58e2e6cceb034df5ba601e4f7a2639d0: Generated new TSK 0
I20260929 19:01:41.594544 10188 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 19:01:41.603899 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUbnLHE/build/release/bin/kudu
/tmp/dist-test-taskUbnLHE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708447806542-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708447806542-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708447806542-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708447806542-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46663
--builtin_ntp_servers=127.0.98.20:33551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260929 19:01:41.683678 10200 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:01:41.683871 10200 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:01:41.683892 10200 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:01:41.685051 10200 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:01:41.685091 10200 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:01:41.685115 10200 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 19:01:41.686368 10200 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708447806542-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708447806542-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708447806542-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46663
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10200
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708447806542-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 18:09:13 UTC on ccycloud.quasar-hpkalf.root.comops.site
build id 4680913
I20260929 19:01:41.686530 10200 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:01:41.686751 10200 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:01:41.691425 10206 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:01:41.691431 10205 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:01:41.691684 10200 server_base.cc:1034] running on GCE node
W20260929 19:01:41.691444 10208 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:01:41.692003 10200 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:01:41.692239 10200 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:01:41.693378 10200 hybrid_clock.cc:648] HybridClock initialized: now 1790708501693358 us; error 38 us; skew 500 ppm
I20260929 19:01:41.694598 10200 webserver.cc:460] Webserver started at http://127.0.98.1:38279/ using document root <none> and password file <none>
I20260929 19:01:41.694842 10200 fs_manager.cc:359] Metadata directory not provided
I20260929 19:01:41.694890 10200 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:01:41.695026 10200 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:01:41.696014 10200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708447806542-392-0/minicluster-data/ts-0/data/instance:
uuid: "100d1dc1272747429b9861fc0c8783e0"
format_stamp: "Formatted at 2026-09-29 19:01:41 on dist-test-slave-lqkv"
server_key: "a2c19ad2fa80cb906fb0ca2ec58f7ecf"
server_key_iv: "8446cbc2bfff5211b94bec4064de8639"
server_key_version: "encryptionkey@0"
I20260929 19:01:41.696367 10200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708447806542-392-0/minicluster-data/ts-0/wal/instance:
uuid: "100d1dc1272747429b9861fc0c8783e0"
format_stamp: "Formatted at 2026-09-29 19:01:41 on dist-test-slave-lqkv"
server_key: "a2c19ad2fa80cb906fb0ca2ec58f7ecf"
server_key_iv: "8446cbc2bfff5211b94bec4064de8639"
server_key_version: "encryptionkey@0"
I20260929 19:01:41.697691 10200 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 19:01:41.698403 10214 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:41.698575 10200 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 19:01:41.698683 10200 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708447806542-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708447806542-392-0/minicluster-data/ts-0/wal
uuid: "100d1dc1272747429b9861fc0c8783e0"
format_stamp: "Formatted at 2026-09-29 19:01:41 on dist-test-slave-lqkv"
server_key: "a2c19ad2fa80cb906fb0ca2ec58f7ecf"
server_key_iv: "8446cbc2bfff5211b94bec4064de8639"
server_key_version: "encryptionkey@0"
I20260929 19:01:41.698767 10200 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708447806542-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708447806542-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708447806542-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:01:41.702028 10200 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:01:41.702268 10200 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:01:41.702399 10200 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:01:41.702598 10200 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:01:41.702886 10200 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:01:41.702939 10200 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:41.702979 10200 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:01:41.703007 10200 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:41.708698 10200 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42777
I20260929 19:01:41.708729 10327 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42777 every 8 connection(s)
I20260929 19:01:41.709057 10200 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708447806542-392-0/minicluster-data/ts-0/data/info.pb
I20260929 19:01:41.711855 10196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:41.709709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40647 (local address 127.0.98.62:46663)
0929 19:01:41.709769 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:41.709771 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:41.709981 (+ 210us) server_negotiation.cc:407] Connection header received
0929 19:01:41.710433 (+ 452us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:41.710435 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:41.710442 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:41.710485 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:01:41.711600 (+ 1115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:01:41.711602 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:01:41.711602 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 19:01:41.711607 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:01:41.711621 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 19:01:41.711626 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:01:41.711676 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0929 19:01:41.711770 (+ 94us) server_negotiation.cc:299] Negotiation successful
0929 19:01:41.711786 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260929 19:01:41.712028 10330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:41.709701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46663 (local address 127.0.98.1:40647)
0929 19:01:41.709877 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:41.709905 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0929 19:01:41.710331 (+ 426us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:01:41.710554 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:01:41.710557 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:01:41.710572 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:01:41.711451 (+ 879us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 19:01:41.711451 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:01:41.711475 (+ 24us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 19:01:41.711499 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:01:41.711684 (+ 185us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:01:41.711688 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:01:41.711691 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 19:01:41.711780 (+ 89us) client_negotiation.cc:240] Negotiation successful
0929 19:01:41.711835 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":80,"threads_started":1}
I20260929 19:01:41.712487 10328 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46663
I20260929 19:01:41.712553 10328 heartbeater.cc:464] Registering TS with master...
I20260929 19:01:41.712695 10328 heartbeater.cc:511] Master 127.0.98.62:46663 requested a full tablet report, sending...
I20260929 19:01:41.713194 10140 ts_manager.cc:194] Registered new tserver with Master: 100d1dc1272747429b9861fc0c8783e0 (127.0.98.1:42777)
I20260929 19:01:41.718101 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUbnLHE/build/release/bin/kudu as pid 10200
I20260929 19:01:41.718184 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708447806542-392-0/minicluster-data/ts-0/wal/instance
I20260929 19:01:41.718375 392 external_mini_cluster.cc:1428] Setting key 88ebb0f8d0aae1ba459ae004efa554e5
I20260929 19:01:41.719168 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUbnLHE/build/release/bin/kudu
/tmp/dist-test-taskUbnLHE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708447806542-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708447806542-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708447806542-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708447806542-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46663
--builtin_ntp_servers=127.0.98.20:33551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260929 19:01:41.795507 10331 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:01:41.795709 10331 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:01:41.795744 10331 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:01:41.796918 10331 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:01:41.796972 10331 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:01:41.797011 10331 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 19:01:41.798283 10331 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708447806542-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708447806542-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708447806542-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46663
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10331
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708447806542-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 18:09:13 UTC on ccycloud.quasar-hpkalf.root.comops.site
build id 4680913
I20260929 19:01:41.798537 10331 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:01:41.798799 10331 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:01:41.803614 10339 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:01:41.803615 10336 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:01:41.803596 10337 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:01:41.803802 10331 server_base.cc:1034] running on GCE node
I20260929 19:01:41.803999 10331 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:01:41.804247 10331 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:01:41.805401 10331 hybrid_clock.cc:648] HybridClock initialized: now 1790708501805392 us; error 31 us; skew 500 ppm
I20260929 19:01:41.806470 10331 webserver.cc:460] Webserver started at http://127.0.98.2:41159/ using document root <none> and password file <none>
I20260929 19:01:41.806695 10331 fs_manager.cc:359] Metadata directory not provided
I20260929 19:01:41.806761 10331 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:01:41.806898 10331 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:01:41.807806 10331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708447806542-392-0/minicluster-data/ts-1/data/instance:
uuid: "070d701fbc784be19c76c07e6d2c13b2"
format_stamp: "Formatted at 2026-09-29 19:01:41 on dist-test-slave-lqkv"
server_key: "3f63dbad00cf3f263247daa305317adb"
server_key_iv: "acb17e1963ff672456121b1761f3adc8"
server_key_version: "encryptionkey@0"
I20260929 19:01:41.808135 10331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708447806542-392-0/minicluster-data/ts-1/wal/instance:
uuid: "070d701fbc784be19c76c07e6d2c13b2"
format_stamp: "Formatted at 2026-09-29 19:01:41 on dist-test-slave-lqkv"
server_key: "3f63dbad00cf3f263247daa305317adb"
server_key_iv: "acb17e1963ff672456121b1761f3adc8"
server_key_version: "encryptionkey@0"
I20260929 19:01:41.809460 10331 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 19:01:41.810068 10345 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:41.810210 10331 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 19:01:41.810292 10331 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708447806542-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708447806542-392-0/minicluster-data/ts-1/wal
uuid: "070d701fbc784be19c76c07e6d2c13b2"
format_stamp: "Formatted at 2026-09-29 19:01:41 on dist-test-slave-lqkv"
server_key: "3f63dbad00cf3f263247daa305317adb"
server_key_iv: "acb17e1963ff672456121b1761f3adc8"
server_key_version: "encryptionkey@0"
I20260929 19:01:41.810366 10331 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708447806542-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708447806542-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708447806542-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:01:41.813522 10331 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:01:41.813762 10331 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:01:41.813910 10331 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:01:41.814105 10331 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:01:41.814388 10331 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:01:41.814438 10331 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:41.814476 10331 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:01:41.814502 10331 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:41.819893 10331 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38823
I20260929 19:01:41.819932 10458 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38823 every 8 connection(s)
I20260929 19:01:41.820192 10331 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708447806542-392-0/minicluster-data/ts-1/data/info.pb
I20260929 19:01:41.822657 10196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:41.820764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36991 (local address 127.0.98.62:46663)
0929 19:01:41.820856 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:41.820858 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:41.821078 (+ 220us) server_negotiation.cc:407] Connection header received
0929 19:01:41.821476 (+ 398us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:41.821477 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:41.821482 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:41.821530 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:01:41.822494 (+ 964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:01:41.822495 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:01:41.822496 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 19:01:41.822499 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:01:41.822513 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 19:01:41.822516 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:01:41.822561 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0929 19:01:41.822600 (+ 39us) server_negotiation.cc:299] Negotiation successful
0929 19:01:41.822610 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 19:01:41.822857 10461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:41.820789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46663 (local address 127.0.98.2:36991)
0929 19:01:41.820973 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:41.821000 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0929 19:01:41.821401 (+ 401us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:01:41.821575 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:01:41.821578 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:01:41.821586 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:01:41.822364 (+ 778us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 19:01:41.822365 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:01:41.822388 (+ 23us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 19:01:41.822410 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:01:41.822557 (+ 147us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:01:41.822559 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:01:41.822561 (+ 2us) client_negotiation.cc:769] Sending connection context
0929 19:01:41.822605 (+ 44us) client_negotiation.cc:240] Negotiation successful
0929 19:01:41.822645 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":111,"threads_started":1}
I20260929 19:01:41.823237 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUbnLHE/build/release/bin/kudu as pid 10331
I20260929 19:01:41.823341 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708447806542-392-0/minicluster-data/ts-1/wal/instance
I20260929 19:01:41.823429 10459 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46663
I20260929 19:01:41.823531 10459 heartbeater.cc:464] Registering TS with master...
I20260929 19:01:41.823544 392 external_mini_cluster.cc:1428] Setting key 1549f1872ae5150c186df0892f1b50f1
I20260929 19:01:41.823666 10459 heartbeater.cc:511] Master 127.0.98.62:46663 requested a full tablet report, sending...
I20260929 19:01:41.824013 10139 ts_manager.cc:194] Registered new tserver with Master: 070d701fbc784be19c76c07e6d2c13b2 (127.0.98.2:38823)
I20260929 19:01:41.824528 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUbnLHE/build/release/bin/kudu
/tmp/dist-test-taskUbnLHE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708447806542-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708447806542-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708447806542-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708447806542-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46663
--builtin_ntp_servers=127.0.98.20:33551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260929 19:01:41.899616 10462 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:01:41.899791 10462 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:01:41.899812 10462 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:01:41.900940 10462 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:01:41.900976 10462 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:01:41.901001 10462 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 19:01:41.902429 10462 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708447806542-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708447806542-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708447806542-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46663
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10462
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708447806542-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 18:09:13 UTC on ccycloud.quasar-hpkalf.root.comops.site
build id 4680913
I20260929 19:01:41.902657 10462 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:01:41.902937 10462 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:01:41.907495 10468 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:01:41.907579 10462 server_base.cc:1034] running on GCE node
W20260929 19:01:41.907474 10470 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:01:41.907481 10467 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:01:41.907835 10462 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:01:41.908010 10462 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:01:41.909135 10462 hybrid_clock.cc:648] HybridClock initialized: now 1790708501909108 us; error 46 us; skew 500 ppm
I20260929 19:01:41.910230 10462 webserver.cc:460] Webserver started at http://127.0.98.3:43453/ using document root <none> and password file <none>
I20260929 19:01:41.910461 10462 fs_manager.cc:359] Metadata directory not provided
I20260929 19:01:41.910503 10462 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:01:41.910657 10462 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:01:41.911582 10462 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708447806542-392-0/minicluster-data/ts-2/data/instance:
uuid: "4bf6d9341a8e4911bca0cd551f2212f1"
format_stamp: "Formatted at 2026-09-29 19:01:41 on dist-test-slave-lqkv"
server_key: "91b7740cfa05230bcd498a7760bebb81"
server_key_iv: "bf51a47d627392fad09766acfaae28a6"
server_key_version: "encryptionkey@0"
I20260929 19:01:41.911907 10462 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708447806542-392-0/minicluster-data/ts-2/wal/instance:
uuid: "4bf6d9341a8e4911bca0cd551f2212f1"
format_stamp: "Formatted at 2026-09-29 19:01:41 on dist-test-slave-lqkv"
server_key: "91b7740cfa05230bcd498a7760bebb81"
server_key_iv: "bf51a47d627392fad09766acfaae28a6"
server_key_version: "encryptionkey@0"
I20260929 19:01:41.913224 10462 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 19:01:41.913928 10476 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:41.914089 10462 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 19:01:41.914168 10462 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708447806542-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708447806542-392-0/minicluster-data/ts-2/wal
uuid: "4bf6d9341a8e4911bca0cd551f2212f1"
format_stamp: "Formatted at 2026-09-29 19:01:41 on dist-test-slave-lqkv"
server_key: "91b7740cfa05230bcd498a7760bebb81"
server_key_iv: "bf51a47d627392fad09766acfaae28a6"
server_key_version: "encryptionkey@0"
I20260929 19:01:41.914228 10462 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708447806542-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708447806542-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708447806542-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:01:41.917258 10462 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:01:41.917447 10462 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:01:41.917539 10462 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:01:41.917675 10462 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:01:41.917904 10462 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:01:41.917932 10462 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:41.917950 10462 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:01:41.917963 10462 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:41.923199 10462 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35107
I20260929 19:01:41.923230 10589 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35107 every 8 connection(s)
I20260929 19:01:41.923539 10462 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708447806542-392-0/minicluster-data/ts-2/data/info.pb
I20260929 19:01:41.926088 10196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:41.924174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52823 (local address 127.0.98.62:46663)
0929 19:01:41.924232 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:41.924233 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:41.924454 (+ 221us) server_negotiation.cc:407] Connection header received
0929 19:01:41.924846 (+ 392us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:41.924848 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:41.924854 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:41.924932 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:01:41.925919 (+ 987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:01:41.925920 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:01:41.925920 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 19:01:41.925924 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:01:41.925937 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 19:01:41.925940 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:01:41.925985 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0929 19:01:41.926029 (+ 44us) server_negotiation.cc:299] Negotiation successful
0929 19:01:41.926041 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260929 19:01:41.926232 10592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:41.924140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46663 (local address 127.0.98.3:52823)
0929 19:01:41.924304 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:41.924339 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0929 19:01:41.924727 (+ 388us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:01:41.924981 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:01:41.924984 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:01:41.924992 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:01:41.925788 (+ 796us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 19:01:41.925789 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:01:41.925812 (+ 23us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 19:01:41.925835 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:01:41.925982 (+ 147us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:01:41.925984 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:01:41.925986 (+ 2us) client_negotiation.cc:769] Sending connection context
0929 19:01:41.926028 (+ 42us) client_negotiation.cc:240] Negotiation successful
0929 19:01:41.926068 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":85,"threads_started":1}
I20260929 19:01:41.926702 10590 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46663
I20260929 19:01:41.926790 10590 heartbeater.cc:464] Registering TS with master...
I20260929 19:01:41.926919 10590 heartbeater.cc:511] Master 127.0.98.62:46663 requested a full tablet report, sending...
I20260929 19:01:41.927263 10139 ts_manager.cc:194] Registered new tserver with Master: 4bf6d9341a8e4911bca0cd551f2212f1 (127.0.98.3:35107)
I20260929 19:01:41.928545 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUbnLHE/build/release/bin/kudu as pid 10462
I20260929 19:01:41.928643 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790708447806542-392-0/minicluster-data/ts-2/wal/instance
I20260929 19:01:41.928797 392 external_mini_cluster.cc:1428] Setting key bb9d5e26d02f0921e763a05d4a9491ab
I20260929 19:01:41.929469 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260929 19:01:41.931028 10196 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:01:41.930616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60402 (local address 127.0.98.62:46663)
0929 19:01:41.930666 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:41.930667 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:41.930792 (+ 125us) server_negotiation.cc:407] Connection header received
0929 19:01:41.930822 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:41.930822 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:41.930825 (+ 3us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:41.930864 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:01:41.930981 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60402: BlockingRecv error: recv got EOF from 127.0.0.1:60402 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
I20260929 19:01:41.931082 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUbnLHE/build/release/bin/kudu with pid 10200
I20260929 19:01:41.936286 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUbnLHE/build/release/bin/kudu with pid 10331
I20260929 19:01:41.940407 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUbnLHE/build/release/bin/kudu with pid 10462
I20260929 19:01:41.944216 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUbnLHE/build/release/bin/kudu with pid 10109
2026-09-29T19:01:41Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (511 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708447806542-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 19:01:41 dist-test-slave-lqkv krb5kdc[10600](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 19:01:41 dist-test-slave-lqkv krb5kdc[10600](info): set up 2 sockets
Sep 29 19:01:41 dist-test-slave-lqkv krb5kdc[10600](info): commencing operation
krb5kdc: starting...
W20260929 19:01:42.981608 392 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708447806542-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708447806542-392-0/krb5kdc/test-user.keytab.
Sep 29 19:01:43 dist-test-slave-lqkv krb5kdc[10600](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708503, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T19:01:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T19:01:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708447806542-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708447806542-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 19:01:43.087574 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUbnLHE/build/release/bin/kudu
/tmp/dist-test-taskUbnLHE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708447806542-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708447806542-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708447806542-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708447806542-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33845
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708447806542-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33845
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708447806542-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708447806542-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708447806542-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:01:43.161103 10613 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:01:43.161319 10613 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:01:43.161355 10613 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:01:43.162518 10613 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 19:01:43.162590 10613 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:01:43.162614 10613 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:01:43.162643 10613 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 19:01:43.162674 10613 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 19:01:43.164111 10613 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708447806542-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708447806542-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33845
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708447806542-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33845
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708447806542-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10613
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708447806542-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 18:09:13 UTC on ccycloud.quasar-hpkalf.root.comops.site
build id 4680913
I20260929 19:01:43.164369 10613 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:01:43.164583 10613 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:01:43.169013 10619 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:01:43.169019 10621 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:01:43.169063 10613 server_base.cc:1034] running on GCE node
W20260929 19:01:43.169068 10618 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:01:43.169402 10613 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:01:43.169679 10613 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:01:43.170811 10613 hybrid_clock.cc:648] HybridClock initialized: now 1790708503170801 us; error 29 us; skew 500 ppm
Sep 29 19:01:43 dist-test-slave-lqkv krb5kdc[10600](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708503, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:01:43.173002 10613 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 19:01:43.173409 10613 webserver.cc:460] Webserver started at http://127.0.98.62:37313/ using document root <none> and password file <none>
I20260929 19:01:43.173636 10613 fs_manager.cc:359] Metadata directory not provided
I20260929 19:01:43.173681 10613 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:01:43.173858 10613 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:01:43.174849 10613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708447806542-392-0/minicluster-data/master-0/data/instance:
uuid: "b0c1f78bc00e4da0a1d58b6ab7878157"
format_stamp: "Formatted at 2026-09-29 19:01:43 on dist-test-slave-lqkv"
server_key: "0bbaecb202ead49f85453d1f9d8bed4f"
server_key_iv: "e3fd0fcb52e15d422514ee654421d13c"
server_key_version: "encryptionkey@0"
I20260929 19:01:43.175202 10613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708447806542-392-0/minicluster-data/master-0/wal/instance:
uuid: "b0c1f78bc00e4da0a1d58b6ab7878157"
format_stamp: "Formatted at 2026-09-29 19:01:43 on dist-test-slave-lqkv"
server_key: "0bbaecb202ead49f85453d1f9d8bed4f"
server_key_iv: "e3fd0fcb52e15d422514ee654421d13c"
server_key_version: "encryptionkey@0"
I20260929 19:01:43.176373 10613 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 19:01:43.177069 10628 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:43.177237 10613 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 19:01:43.177325 10613 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708447806542-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708447806542-392-0/minicluster-data/master-0/wal
uuid: "b0c1f78bc00e4da0a1d58b6ab7878157"
format_stamp: "Formatted at 2026-09-29 19:01:43 on dist-test-slave-lqkv"
server_key: "0bbaecb202ead49f85453d1f9d8bed4f"
server_key_iv: "e3fd0fcb52e15d422514ee654421d13c"
server_key_version: "encryptionkey@0"
I20260929 19:01:43.177397 10613 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708447806542-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708447806542-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708447806542-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:01:43.193076 10613 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:01:43.195222 10613 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:01:43.195375 10613 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:01:43.199254 10613 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33845
I20260929 19:01:43.199285 10680 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33845 every 8 connection(s)
I20260929 19:01:43.199589 10613 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708447806542-392-0/minicluster-data/master-0/data/info.pb
I20260929 19:01:43.200466 10681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:01:43.201246 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUbnLHE/build/release/bin/kudu as pid 10613
I20260929 19:01:43.201325 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708447806542-392-0/minicluster-data/master-0/wal/instance
I20260929 19:01:43.201509 392 external_mini_cluster.cc:1428] Setting key 2190c69828c0feb5af6f1735b7a1c765
I20260929 19:01:43.202826 10681 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0c1f78bc00e4da0a1d58b6ab7878157: Bootstrap starting.
I20260929 19:01:43.203481 10681 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b0c1f78bc00e4da0a1d58b6ab7878157: Neither blocks nor log segments found. Creating new log.
I20260929 19:01:43.203799 10681 log.cc:824] T 00000000000000000000000000000000 P b0c1f78bc00e4da0a1d58b6ab7878157: Log is configured to *not* fsync() on all Append() calls
I20260929 19:01:43.204432 10681 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0c1f78bc00e4da0a1d58b6ab7878157: No bootstrap required, opened a new log
I20260929 19:01:43.206120 10681 raft_consensus.cc:348] T 00000000000000000000000000000000 P b0c1f78bc00e4da0a1d58b6ab7878157 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0c1f78bc00e4da0a1d58b6ab7878157" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33845 } }
I20260929 19:01:43.206259 10681 raft_consensus.cc:374] T 00000000000000000000000000000000 P b0c1f78bc00e4da0a1d58b6ab7878157 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:01:43.206303 10681 raft_consensus.cc:729] T 00000000000000000000000000000000 P b0c1f78bc00e4da0a1d58b6ab7878157 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0c1f78bc00e4da0a1d58b6ab7878157, State: Initialized, Role: FOLLOWER
I20260929 19:01:43.206429 10681 consensus_queue.cc:260] T 00000000000000000000000000000000 P b0c1f78bc00e4da0a1d58b6ab7878157 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0c1f78bc00e4da0a1d58b6ab7878157" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33845 } }
I20260929 19:01:43.206530 10681 raft_consensus.cc:388] T 00000000000000000000000000000000 P b0c1f78bc00e4da0a1d58b6ab7878157 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 19:01:43.206571 10681 raft_consensus.cc:482] T 00000000000000000000000000000000 P b0c1f78bc00e4da0a1d58b6ab7878157 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 19:01:43.206619 10681 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b0c1f78bc00e4da0a1d58b6ab7878157 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:01:43.207073 10681 raft_consensus.cc:504] T 00000000000000000000000000000000 P b0c1f78bc00e4da0a1d58b6ab7878157 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0c1f78bc00e4da0a1d58b6ab7878157" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33845 } }
I20260929 19:01:43.207185 10681 leader_election.cc:302] T 00000000000000000000000000000000 P b0c1f78bc00e4da0a1d58b6ab7878157 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b0c1f78bc00e4da0a1d58b6ab7878157; no voters:
I20260929 19:01:43.207322 10681 leader_election.cc:288] T 00000000000000000000000000000000 P b0c1f78bc00e4da0a1d58b6ab7878157 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 19:01:43.207363 10686 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b0c1f78bc00e4da0a1d58b6ab7878157 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:01:43.207504 10686 raft_consensus.cc:686] T 00000000000000000000000000000000 P b0c1f78bc00e4da0a1d58b6ab7878157 [term 1 LEADER]: Becoming Leader. State: Replica: b0c1f78bc00e4da0a1d58b6ab7878157, State: Running, Role: LEADER
I20260929 19:01:43.207562 10681 sys_catalog.cc:564] T 00000000000000000000000000000000 P b0c1f78bc00e4da0a1d58b6ab7878157 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 19:01:43.207567 10686 consensus_queue.cc:237] T 00000000000000000000000000000000 P b0c1f78bc00e4da0a1d58b6ab7878157 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0c1f78bc00e4da0a1d58b6ab7878157" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33845 } }
Sep 29 19:01:43 dist-test-slave-lqkv krb5kdc[10600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708503, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 19:01:43.207844 10687 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0c1f78bc00e4da0a1d58b6ab7878157 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b0c1f78bc00e4da0a1d58b6ab7878157" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0c1f78bc00e4da0a1d58b6ab7878157" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33845 } } }
I20260929 19:01:43.207911 10687 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0c1f78bc00e4da0a1d58b6ab7878157 [sys.catalog]: This master's current role is: LEADER
I20260929 19:01:43.208134 10691 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 19:01:43.208176 10688 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0c1f78bc00e4da0a1d58b6ab7878157 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b0c1f78bc00e4da0a1d58b6ab7878157. Latest consensus state: current_term: 1 leader_uuid: "b0c1f78bc00e4da0a1d58b6ab7878157" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0c1f78bc00e4da0a1d58b6ab7878157" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33845 } } }
I20260929 19:01:43.208690 10691 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 19:01:43.209605 10688 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0c1f78bc00e4da0a1d58b6ab7878157 [sys.catalog]: This master's current role is: LEADER
I20260929 19:01:43.210505 10691 catalog_manager.cc:1347] Generated new cluster ID: edafe0287f4641c0acac5a4e93052fc0
I20260929 19:01:43.210561 10691 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 19:01:43.211930 10684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:43.202006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37874 (local address 127.0.98.62:33845)
0929 19:01:43.202169 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:43.202176 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:43.202198 (+ 22us) server_negotiation.cc:407] Connection header received
0929 19:01:43.202621 (+ 423us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:43.202626 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:43.202666 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:43.202768 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:01:43.203560 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:43.204514 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:43.205331 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:43.205712 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:43.205823 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:43.208322 (+ 2499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:01:43.208326 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:01:43.208327 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:01:43.208339 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:01:43.210252 (+ 1913us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:43.210629 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:43.210632 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:43.210632 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:43.210733 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:43.210905 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:43.210905 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:43.210906 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:43.211165 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:01:43.211277 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0929 19:01:43.211649 (+ 372us) server_negotiation.cc:299] Negotiation successful
0929 19:01:43.211762 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":91,"threads_started":1}
I20260929 19:01:43.243373 10691 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 19:01:43.243901 10691 catalog_manager.cc:1504] Loading token signing keys...
I20260929 19:01:43.248339 10691 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b0c1f78bc00e4da0a1d58b6ab7878157: Generated new TSK 0
I20260929 19:01:43.248500 10691 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708447806542-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708447806542-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260929 19:01:43.276335 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUbnLHE/build/release/bin/kudu
/tmp/dist-test-taskUbnLHE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708447806542-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708447806542-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708447806542-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708447806542-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33845
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708447806542-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708447806542-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708447806542-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708447806542-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:01:43.352269 10707 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:01:43.352452 10707 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:01:43.352475 10707 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:01:43.353657 10707 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:01:43.353698 10707 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:01:43.353724 10707 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 19:01:43.355118 10707 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708447806542-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708447806542-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708447806542-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708447806542-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33845
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10707
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708447806542-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 18:09:13 UTC on ccycloud.quasar-hpkalf.root.comops.site
build id 4680913
I20260929 19:01:43.355336 10707 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:01:43.355571 10707 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:01:43.360074 10713 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:01:43.360271 10715 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:01:43.360081 10712 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:01:43.360268 10707 server_base.cc:1034] running on GCE node
I20260929 19:01:43.360517 10707 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:01:43.360719 10707 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:01:43.361851 10707 hybrid_clock.cc:648] HybridClock initialized: now 1790708503361844 us; error 40 us; skew 500 ppm
Sep 29 19:01:43 dist-test-slave-lqkv krb5kdc[10600](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708503, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:01:43.364254 10707 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 19:01:43.364688 10707 webserver.cc:460] Webserver started at http://127.0.98.1:37219/ using document root <none> and password file <none>
I20260929 19:01:43.364929 10707 fs_manager.cc:359] Metadata directory not provided
I20260929 19:01:43.364984 10707 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:01:43.365121 10707 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:01:43.366016 10707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708447806542-392-0/minicluster-data/ts-0/data/instance:
uuid: "ce010af0e43140678ac8b9cb7d252266"
format_stamp: "Formatted at 2026-09-29 19:01:43 on dist-test-slave-lqkv"
server_key: "5248d7b1b12819c3177a4c73089b34f1"
server_key_iv: "1d7577de686f231c65a8da3e2bb4ab4c"
server_key_version: "encryptionkey@0"
I20260929 19:01:43.366329 10707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708447806542-392-0/minicluster-data/ts-0/wal/instance:
uuid: "ce010af0e43140678ac8b9cb7d252266"
format_stamp: "Formatted at 2026-09-29 19:01:43 on dist-test-slave-lqkv"
server_key: "5248d7b1b12819c3177a4c73089b34f1"
server_key_iv: "1d7577de686f231c65a8da3e2bb4ab4c"
server_key_version: "encryptionkey@0"
I20260929 19:01:43.367534 10707 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260929 19:01:43.368173 10722 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:43.368335 10707 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 19:01:43.368424 10707 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708447806542-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708447806542-392-0/minicluster-data/ts-0/wal
uuid: "ce010af0e43140678ac8b9cb7d252266"
format_stamp: "Formatted at 2026-09-29 19:01:43 on dist-test-slave-lqkv"
server_key: "5248d7b1b12819c3177a4c73089b34f1"
server_key_iv: "1d7577de686f231c65a8da3e2bb4ab4c"
server_key_version: "encryptionkey@0"
I20260929 19:01:43.368500 10707 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708447806542-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708447806542-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708447806542-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:01:43.378000 10707 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:01:43.380316 10707 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:01:43.380465 10707 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:01:43.380681 10707 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:01:43.380975 10707 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:01:43.381006 10707 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:43.381026 10707 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:01:43.381037 10707 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:43.386600 10707 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45037
I20260929 19:01:43.386646 10835 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45037 every 8 connection(s)
I20260929 19:01:43.386919 10707 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708447806542-392-0/minicluster-data/ts-0/data/info.pb
I20260929 19:01:43.390437 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUbnLHE/build/release/bin/kudu as pid 10707
I20260929 19:01:43.390533 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708447806542-392-0/minicluster-data/ts-0/wal/instance
I20260929 19:01:43.390698 392 external_mini_cluster.cc:1428] Setting key 7862fd9b9b0233e93d50665922b11edb
Sep 29 19:01:43 dist-test-slave-lqkv krb5kdc[10600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708503, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 19:01:43.395758 10684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:43.387471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35095 (local address 127.0.98.62:33845)
0929 19:01:43.387546 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:43.387548 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:43.387709 (+ 161us) server_negotiation.cc:407] Connection header received
0929 19:01:43.388162 (+ 453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:43.388163 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:43.388174 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:43.388233 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:01:43.389114 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:43.389568 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:43.390488 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:43.390750 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:43.390831 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:43.393052 (+ 2221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:01:43.393053 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:01:43.393054 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:01:43.393062 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:01:43.394372 (+ 1310us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:43.394865 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:43.394865 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:43.394866 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:43.394914 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:43.395186 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:43.395187 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:43.395187 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:43.395281 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:01:43.395328 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0929 19:01:43.395632 (+ 304us) server_negotiation.cc:299] Negotiation successful
0929 19:01:43.395702 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 19:01:43.396035 10838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:43.387437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33845 (local address 127.0.98.1:35095)
0929 19:01:43.387586 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:43.387617 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0929 19:01:43.388061 (+ 444us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:01:43.388292 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:01:43.388295 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:01:43.388811 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:01:43.389027 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:43.389030 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:43.389672 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:43.389673 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:43.390401 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:43.390403 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:43.390838 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:43.390840 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:43.390977 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:43.391447 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:01:43.391454 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:01:43.392970 (+ 1516us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:01:43.394450 (+ 1480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:01:43.394456 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:01:43.394458 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:01:43.394783 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:01:43.394991 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:01:43.394991 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:01:43.394992 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:01:43.395119 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:01:43.395343 (+ 224us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:01:43.395348 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:01:43.395500 (+ 152us) client_negotiation.cc:769] Sending connection context
0929 19:01:43.395625 (+ 125us) client_negotiation.cc:240] Negotiation successful
0929 19:01:43.395786 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":92,"threads_started":1}
I20260929 19:01:43.396533 10836 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33845
I20260929 19:01:43.396610 10836 heartbeater.cc:464] Registering TS with master...
I20260929 19:01:43.396759 10836 heartbeater.cc:511] Master 127.0.98.62:33845 requested a full tablet report, sending...
I20260929 19:01:43.397229 10645 ts_manager.cc:194] Registered new tserver with Master: ce010af0e43140678ac8b9cb7d252266 (127.0.98.1:45037)
I20260929 19:01:43.397804 10645 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:35095
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708447806542-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708447806542-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260929 19:01:43.413336 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUbnLHE/build/release/bin/kudu
/tmp/dist-test-taskUbnLHE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708447806542-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708447806542-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708447806542-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708447806542-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33845
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708447806542-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708447806542-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708447806542-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708447806542-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:01:43.487741 10841 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:01:43.487905 10841 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:01:43.487926 10841 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:01:43.488970 10841 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:01:43.489004 10841 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:01:43.489028 10841 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 19:01:43.490322 10841 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708447806542-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708447806542-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708447806542-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708447806542-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33845
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10841
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708447806542-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 18:09:13 UTC on ccycloud.quasar-hpkalf.root.comops.site
build id 4680913
I20260929 19:01:43.490505 10841 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:01:43.490649 10841 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:01:43.495178 10849 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:01:43.495188 10846 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:01:43.495266 10841 server_base.cc:1034] running on GCE node
W20260929 19:01:43.495236 10847 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:01:43.495658 10841 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:01:43.495908 10841 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:01:43.497057 10841 hybrid_clock.cc:648] HybridClock initialized: now 1790708503497043 us; error 25 us; skew 500 ppm
Sep 29 19:01:43 dist-test-slave-lqkv krb5kdc[10600](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708503, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:01:43.499202 10841 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 19:01:43.499557 10841 webserver.cc:460] Webserver started at http://127.0.98.2:36249/ using document root <none> and password file <none>
I20260929 19:01:43.499769 10841 fs_manager.cc:359] Metadata directory not provided
I20260929 19:01:43.499827 10841 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:01:43.499949 10841 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:01:43.500787 10841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708447806542-392-0/minicluster-data/ts-1/data/instance:
uuid: "66d15106dd5746d6a1ab2074ad38fbff"
format_stamp: "Formatted at 2026-09-29 19:01:43 on dist-test-slave-lqkv"
server_key: "fb938ffdb9c5462115c0c0ac1640f34d"
server_key_iv: "e8efacde407169e1b8f2541a40ebb459"
server_key_version: "encryptionkey@0"
I20260929 19:01:43.501147 10841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708447806542-392-0/minicluster-data/ts-1/wal/instance:
uuid: "66d15106dd5746d6a1ab2074ad38fbff"
format_stamp: "Formatted at 2026-09-29 19:01:43 on dist-test-slave-lqkv"
server_key: "fb938ffdb9c5462115c0c0ac1640f34d"
server_key_iv: "e8efacde407169e1b8f2541a40ebb459"
server_key_version: "encryptionkey@0"
I20260929 19:01:43.502293 10841 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 19:01:43.502910 10856 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:43.503075 10841 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 19:01:43.503166 10841 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708447806542-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708447806542-392-0/minicluster-data/ts-1/wal
uuid: "66d15106dd5746d6a1ab2074ad38fbff"
format_stamp: "Formatted at 2026-09-29 19:01:43 on dist-test-slave-lqkv"
server_key: "fb938ffdb9c5462115c0c0ac1640f34d"
server_key_iv: "e8efacde407169e1b8f2541a40ebb459"
server_key_version: "encryptionkey@0"
I20260929 19:01:43.503252 10841 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708447806542-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708447806542-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708447806542-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:01:43.515702 10841 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:01:43.517407 10841 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:01:43.517586 10841 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:01:43.517802 10841 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:01:43.518072 10841 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:01:43.518126 10841 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:43.518157 10841 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:01:43.518182 10841 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:43.523530 10841 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46739
I20260929 19:01:43.523597 10969 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46739 every 8 connection(s)
I20260929 19:01:43.523881 10841 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708447806542-392-0/minicluster-data/ts-1/data/info.pb
I20260929 19:01:43.527689 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUbnLHE/build/release/bin/kudu as pid 10841
I20260929 19:01:43.527791 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708447806542-392-0/minicluster-data/ts-1/wal/instance
I20260929 19:01:43.527988 392 external_mini_cluster.cc:1428] Setting key d1b9a5d793ef6c0b3feaea863c6ad967
Sep 29 19:01:43 dist-test-slave-lqkv krb5kdc[10600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708503, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 19:01:43.533178 10684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:43.524589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41317 (local address 127.0.98.62:33845)
0929 19:01:43.524683 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:43.524685 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:43.524868 (+ 183us) server_negotiation.cc:407] Connection header received
0929 19:01:43.525345 (+ 477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:43.525346 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:43.525357 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:43.525412 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:01:43.526246 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:43.526726 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:43.527668 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:43.527941 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:43.528025 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:43.530390 (+ 2365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:01:43.530391 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:01:43.530392 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:01:43.530401 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:01:43.531773 (+ 1372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:43.532252 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:43.532253 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:43.532254 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:43.532311 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:43.532558 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:43.532559 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:43.532560 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:43.532666 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:01:43.532745 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0929 19:01:43.533027 (+ 282us) server_negotiation.cc:299] Negotiation successful
0929 19:01:43.533098 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 19:01:43.533355 10972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:43.524555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33845 (local address 127.0.98.2:41317)
0929 19:01:43.524737 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:43.524768 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0929 19:01:43.525246 (+ 478us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:01:43.525479 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:01:43.525482 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:01:43.525987 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:01:43.526151 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:43.526155 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:43.526818 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:43.526819 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:43.527580 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:43.527581 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:43.528028 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:43.528028 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:43.528169 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:43.528721 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:01:43.528732 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:01:43.530286 (+ 1554us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:01:43.531869 (+ 1583us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:01:43.531875 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:01:43.531877 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:01:43.532162 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:01:43.532377 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:01:43.532378 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:01:43.532378 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:01:43.532483 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:01:43.532737 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:01:43.532742 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:01:43.532902 (+ 160us) client_negotiation.cc:769] Sending connection context
0929 19:01:43.533006 (+ 104us) client_negotiation.cc:240] Negotiation successful
0929 19:01:43.533137 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":81,"threads_started":1}
I20260929 19:01:43.533752 10970 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33845
I20260929 19:01:43.533831 10970 heartbeater.cc:464] Registering TS with master...
I20260929 19:01:43.534016 10970 heartbeater.cc:511] Master 127.0.98.62:33845 requested a full tablet report, sending...
I20260929 19:01:43.534369 10645 ts_manager.cc:194] Registered new tserver with Master: 66d15106dd5746d6a1ab2074ad38fbff (127.0.98.2:46739)
I20260929 19:01:43.534741 10645 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:41317
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708447806542-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708447806542-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260929 19:01:43.551515 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUbnLHE/build/release/bin/kudu
/tmp/dist-test-taskUbnLHE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708447806542-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708447806542-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708447806542-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708447806542-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33845
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708447806542-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708447806542-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708447806542-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708447806542-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:01:43.627460 10975 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:01:43.627664 10975 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:01:43.627699 10975 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:01:43.628762 10975 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:01:43.628840 10975 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:01:43.628886 10975 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 19:01:43.630318 10975 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708447806542-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708447806542-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708447806542-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708447806542-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33845
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10975
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708447806542-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 18:09:13 UTC on ccycloud.quasar-hpkalf.root.comops.site
build id 4680913
I20260929 19:01:43.630589 10975 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:01:43.630810 10975 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:01:43.635867 10981 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:01:43.635901 10975 server_base.cc:1034] running on GCE node
W20260929 19:01:43.635921 10983 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:01:43.635875 10980 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:01:43.636165 10975 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:01:43.636346 10975 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:01:43.637533 10975 hybrid_clock.cc:648] HybridClock initialized: now 1790708503637513 us; error 31 us; skew 500 ppm
Sep 29 19:01:43 dist-test-slave-lqkv krb5kdc[10600](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708503, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:01:43.639858 10975 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 19:01:43.640192 10975 webserver.cc:460] Webserver started at http://127.0.98.3:42123/ using document root <none> and password file <none>
I20260929 19:01:43.640344 10975 fs_manager.cc:359] Metadata directory not provided
I20260929 19:01:43.640379 10975 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:01:43.640451 10975 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:01:43.641644 10975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708447806542-392-0/minicluster-data/ts-2/data/instance:
uuid: "5a95da8804ee41d6a672ee947fab39df"
format_stamp: "Formatted at 2026-09-29 19:01:43 on dist-test-slave-lqkv"
server_key: "6ce83187a322fdd50ac2780154586d82"
server_key_iv: "3bd7c649ce8197bc7f55bb5ec36df1d8"
server_key_version: "encryptionkey@0"
I20260929 19:01:43.641988 10975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708447806542-392-0/minicluster-data/ts-2/wal/instance:
uuid: "5a95da8804ee41d6a672ee947fab39df"
format_stamp: "Formatted at 2026-09-29 19:01:43 on dist-test-slave-lqkv"
server_key: "6ce83187a322fdd50ac2780154586d82"
server_key_iv: "3bd7c649ce8197bc7f55bb5ec36df1d8"
server_key_version: "encryptionkey@0"
I20260929 19:01:43.643826 10975 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260929 19:01:43.644898 10990 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:43.645083 10975 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 19:01:43.645197 10975 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708447806542-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708447806542-392-0/minicluster-data/ts-2/wal
uuid: "5a95da8804ee41d6a672ee947fab39df"
format_stamp: "Formatted at 2026-09-29 19:01:43 on dist-test-slave-lqkv"
server_key: "6ce83187a322fdd50ac2780154586d82"
server_key_iv: "3bd7c649ce8197bc7f55bb5ec36df1d8"
server_key_version: "encryptionkey@0"
I20260929 19:01:43.645278 10975 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708447806542-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708447806542-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708447806542-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:01:43.664407 10975 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:01:43.668968 10975 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:01:43.669150 10975 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:01:43.669431 10975 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:01:43.669866 10975 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:01:43.669955 10975 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:43.670001 10975 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:01:43.670023 10975 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:01:43.678483 10975 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36111
I20260929 19:01:43.678519 11103 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36111 every 8 connection(s)
I20260929 19:01:43.678915 10975 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708447806542-392-0/minicluster-data/ts-2/data/info.pb
I20260929 19:01:43.685509 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUbnLHE/build/release/bin/kudu as pid 10975
I20260929 19:01:43.685618 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708447806542-392-0/minicluster-data/ts-2/wal/instance
I20260929 19:01:43.685817 392 external_mini_cluster.cc:1428] Setting key 46c21bad8908d7ff20e8522b7e7247a8
Sep 29 19:01:43 dist-test-slave-lqkv krb5kdc[10600](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708503, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 19:01:43.690953 10684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:43.679823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41151 (local address 127.0.98.62:33845)
0929 19:01:43.679871 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:43.679873 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:43.679886 (+ 13us) server_negotiation.cc:407] Connection header received
0929 19:01:43.680573 (+ 687us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:43.680575 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:43.680588 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:43.680656 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:01:43.681940 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:43.682810 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:43.684341 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:43.684681 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:43.684765 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:43.687480 (+ 2715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:01:43.687481 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:01:43.687482 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:01:43.687490 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:01:43.689290 (+ 1800us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:43.689826 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:43.689827 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:43.689828 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:43.689894 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:43.690209 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:43.690210 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:43.690210 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:43.690354 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:01:43.690425 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 19:01:43.690765 (+ 340us) server_negotiation.cc:299] Negotiation successful
0929 19:01:43.690854 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260929 19:01:43.691179 11106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:43.679587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33845 (local address 127.0.98.3:41151)
0929 19:01:43.679757 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:43.679802 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 19:01:43.680464 (+ 662us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:01:43.680726 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:01:43.680730 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:01:43.681565 (+ 835us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:01:43.681836 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:43.681842 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:43.682917 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:43.682918 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:43.684246 (+ 1328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:43.684248 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:43.684777 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:43.684778 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:43.684966 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:43.685465 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:01:43.685475 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:01:43.687382 (+ 1907us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:01:43.689376 (+ 1994us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:01:43.689383 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:01:43.689385 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:01:43.689728 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:01:43.689976 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:01:43.689977 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:01:43.689977 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:01:43.690126 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:01:43.690432 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:01:43.690438 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:01:43.690608 (+ 170us) client_negotiation.cc:769] Sending connection context
0929 19:01:43.690731 (+ 123us) client_negotiation.cc:240] Negotiation successful
0929 19:01:43.690923 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":76,"threads_started":1}
I20260929 19:01:43.693347 11104 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33845
I20260929 19:01:43.693440 11104 heartbeater.cc:464] Registering TS with master...
I20260929 19:01:43.693605 11104 heartbeater.cc:511] Master 127.0.98.62:33845 requested a full tablet report, sending...
I20260929 19:01:43.693966 10645 ts_manager.cc:194] Registered new tserver with Master: 5a95da8804ee41d6a672ee947fab39df (127.0.98.3:36111)
I20260929 19:01:43.694418 10645 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:41151
I20260929 19:01:43.697238 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 19:01:43.704035 10684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:43.698468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37880 (local address 127.0.98.62:33845)
0929 19:01:43.698519 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:43.698521 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:43.698645 (+ 124us) server_negotiation.cc:407] Connection header received
0929 19:01:43.698734 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:43.698735 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:43.698760 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:43.698826 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:01:43.699372 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:43.699859 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:43.700559 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:43.700745 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:43.700823 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:43.701871 (+ 1048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:01:43.701872 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:01:43.701873 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:01:43.701879 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:01:43.702922 (+ 1043us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:43.703250 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:43.703251 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:43.703252 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:43.703313 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:01:43.703523 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:01:43.703523 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:01:43.703524 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:01:43.703652 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:01:43.703757 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0929 19:01:43.703869 (+ 112us) server_negotiation.cc:299] Negotiation successful
0929 19:01:43.703939 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260929 19:01:43.705427 10645 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37880:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 19:01:43.706982 10645 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 19:01:43.713260 11114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:43.709077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45037 (local address 127.0.0.1:58812)
0929 19:01:43.709515 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:43.709546 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0929 19:01:43.709631 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:01:43.709974 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:01:43.709974 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:01:43.709976 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:01:43.710025 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:43.710027 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:43.710942 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:43.710943 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:43.712187 (+ 1244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:43.712189 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:43.712928 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:43.712929 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:43.713003 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:43.713015 (+ 12us) client_negotiation.cc:769] Sending connection context
0929 19:01:43.713113 (+ 98us) client_negotiation.cc:240] Negotiation successful
0929 19:01:43.713174 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":49,"threads_started":1}
I20260929 19:01:43.713280 11115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:43.709243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58812 (local address 127.0.98.1:45037)
0929 19:01:43.709461 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:43.709465 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:43.709623 (+ 158us) server_negotiation.cc:407] Connection header received
0929 19:01:43.709776 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:43.709779 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:43.709795 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:43.709847 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:01:43.710100 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:43.710824 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:43.712281 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:43.712847 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:43.712922 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:43.712986 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0929 19:01:43.713143 (+ 157us) server_negotiation.cc:299] Negotiation successful
0929 19:01:43.713184 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":68,"threads_started":1}
I20260929 19:01:43.713601 11116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:43.709333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40240 (local address 127.0.98.3:36111)
0929 19:01:43.709495 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:43.709500 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:43.709817 (+ 317us) server_negotiation.cc:407] Connection header received
0929 19:01:43.709851 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:43.709853 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:43.709865 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:43.709933 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:01:43.710106 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:43.711121 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:43.712342 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:43.713051 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:43.713120 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:43.713185 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0929 19:01:43.713468 (+ 283us) server_negotiation.cc:299] Negotiation successful
0929 19:01:43.713521 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":94,"threads_started":1}
I20260929 19:01:43.713652 11117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:43.709379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50024 (local address 127.0.98.2:46739)
0929 19:01:43.709529 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:43.709533 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:43.709600 (+ 67us) server_negotiation.cc:407] Connection header received
0929 19:01:43.709696 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:43.709698 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:43.709713 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:43.709960 (+ 247us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:01:43.710027 (+ 67us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:43.711123 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:43.712400 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:43.713111 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:43.713418 (+ 307us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:43.713477 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0929 19:01:43.713500 (+ 23us) server_negotiation.cc:299] Negotiation successful
0929 19:01:43.713548 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":90,"threads_started":1}
I20260929 19:01:43.713521 11113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:43.709077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46739 (local address 127.0.0.1:50024)
0929 19:01:43.709546 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:43.709556 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 19:01:43.709621 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:01:43.709826 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:01:43.709829 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:01:43.709831 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:01:43.709889 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:43.709891 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:43.711225 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:43.711225 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:43.712316 (+ 1091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:43.712318 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:43.713240 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:43.713241 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:43.713299 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:43.713302 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 19:01:43.713333 (+ 31us) client_negotiation.cc:240] Negotiation successful
0929 19:01:43.713375 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"mutex_wait_us":33,"thread_start_us":47,"threads_started":1}
I20260929 19:01:43.713801 11112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:43.709246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36111 (local address 127.0.0.1:40240)
0929 19:01:43.709712 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:43.709720 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0929 19:01:43.709762 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:01:43.709993 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:01:43.709993 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:01:43.709995 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:01:43.710034 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:43.710036 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:43.711238 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:43.711239 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:43.712267 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:43.712270 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:43.713236 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:43.713237 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:43.713302 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:43.713304 (+ 2us) client_negotiation.cc:769] Sending connection context
0929 19:01:43.713350 (+ 46us) client_negotiation.cc:240] Negotiation successful
0929 19:01:43.713377 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"mutex_wait_us":2,"thread_start_us":43,"threads_started":1}
I20260929 19:01:43.714530 10904 tablet_service.cc:1467] Processing CreateTablet for tablet fa72812a7b4e4340a268b6e495be4cbe (DEFAULT_TABLE table=test-table [id=9dd00c2179b74ae19a27ce9859f0dddd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:01:43.714864 10904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa72812a7b4e4340a268b6e495be4cbe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:01:43.715130 11038 tablet_service.cc:1467] Processing CreateTablet for tablet fa72812a7b4e4340a268b6e495be4cbe (DEFAULT_TABLE table=test-table [id=9dd00c2179b74ae19a27ce9859f0dddd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:01:43.715569 11038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa72812a7b4e4340a268b6e495be4cbe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:01:43.717206 10770 tablet_service.cc:1467] Processing CreateTablet for tablet fa72812a7b4e4340a268b6e495be4cbe (DEFAULT_TABLE table=test-table [id=9dd00c2179b74ae19a27ce9859f0dddd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:01:43.717531 10770 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa72812a7b4e4340a268b6e495be4cbe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:01:43.718379 11118 tablet_bootstrap.cc:492] T fa72812a7b4e4340a268b6e495be4cbe P 66d15106dd5746d6a1ab2074ad38fbff: Bootstrap starting.
I20260929 19:01:43.718945 11118 tablet_bootstrap.cc:654] T fa72812a7b4e4340a268b6e495be4cbe P 66d15106dd5746d6a1ab2074ad38fbff: Neither blocks nor log segments found. Creating new log.
I20260929 19:01:43.719365 11118 log.cc:824] T fa72812a7b4e4340a268b6e495be4cbe P 66d15106dd5746d6a1ab2074ad38fbff: Log is configured to *not* fsync() on all Append() calls
I20260929 19:01:43.719957 11118 tablet_bootstrap.cc:492] T fa72812a7b4e4340a268b6e495be4cbe P 66d15106dd5746d6a1ab2074ad38fbff: No bootstrap required, opened a new log
I20260929 19:01:43.720010 11118 ts_tablet_manager.cc:1397] T fa72812a7b4e4340a268b6e495be4cbe P 66d15106dd5746d6a1ab2074ad38fbff: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 19:01:43.720325 11120 tablet_bootstrap.cc:492] T fa72812a7b4e4340a268b6e495be4cbe P ce010af0e43140678ac8b9cb7d252266: Bootstrap starting.
I20260929 19:01:43.721019 11120 tablet_bootstrap.cc:654] T fa72812a7b4e4340a268b6e495be4cbe P ce010af0e43140678ac8b9cb7d252266: Neither blocks nor log segments found. Creating new log.
I20260929 19:01:43.721560 11120 log.cc:824] T fa72812a7b4e4340a268b6e495be4cbe P ce010af0e43140678ac8b9cb7d252266: Log is configured to *not* fsync() on all Append() calls
I20260929 19:01:43.722227 11122 tablet_bootstrap.cc:492] T fa72812a7b4e4340a268b6e495be4cbe P 5a95da8804ee41d6a672ee947fab39df: Bootstrap starting.
I20260929 19:01:43.722353 11120 tablet_bootstrap.cc:492] T fa72812a7b4e4340a268b6e495be4cbe P ce010af0e43140678ac8b9cb7d252266: No bootstrap required, opened a new log
I20260929 19:01:43.722425 11120 ts_tablet_manager.cc:1397] T fa72812a7b4e4340a268b6e495be4cbe P ce010af0e43140678ac8b9cb7d252266: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 19:01:43.722626 11118 raft_consensus.cc:348] T fa72812a7b4e4340a268b6e495be4cbe P 66d15106dd5746d6a1ab2074ad38fbff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce010af0e43140678ac8b9cb7d252266" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45037 } } peers { permanent_uuid: "5a95da8804ee41d6a672ee947fab39df" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36111 } } peers { permanent_uuid: "66d15106dd5746d6a1ab2074ad38fbff" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46739 } }
I20260929 19:01:43.722796 11118 raft_consensus.cc:374] T fa72812a7b4e4340a268b6e495be4cbe P 66d15106dd5746d6a1ab2074ad38fbff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:01:43.722829 11118 raft_consensus.cc:729] T fa72812a7b4e4340a268b6e495be4cbe P 66d15106dd5746d6a1ab2074ad38fbff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66d15106dd5746d6a1ab2074ad38fbff, State: Initialized, Role: FOLLOWER
I20260929 19:01:43.722885 11122 tablet_bootstrap.cc:654] T fa72812a7b4e4340a268b6e495be4cbe P 5a95da8804ee41d6a672ee947fab39df: Neither blocks nor log segments found. Creating new log.
I20260929 19:01:43.722965 11118 consensus_queue.cc:260] T fa72812a7b4e4340a268b6e495be4cbe P 66d15106dd5746d6a1ab2074ad38fbff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce010af0e43140678ac8b9cb7d252266" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45037 } } peers { permanent_uuid: "5a95da8804ee41d6a672ee947fab39df" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36111 } } peers { permanent_uuid: "66d15106dd5746d6a1ab2074ad38fbff" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46739 } }
I20260929 19:01:43.723232 11118 ts_tablet_manager.cc:1428] T fa72812a7b4e4340a268b6e495be4cbe P 66d15106dd5746d6a1ab2074ad38fbff: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260929 19:01:43.723393 10970 heartbeater.cc:503] Master 127.0.98.62:33845 was elected leader, sending a full tablet report...
I20260929 19:01:43.723469 11122 log.cc:824] T fa72812a7b4e4340a268b6e495be4cbe P 5a95da8804ee41d6a672ee947fab39df: Log is configured to *not* fsync() on all Append() calls
I20260929 19:01:43.724181 11122 tablet_bootstrap.cc:492] T fa72812a7b4e4340a268b6e495be4cbe P 5a95da8804ee41d6a672ee947fab39df: No bootstrap required, opened a new log
I20260929 19:01:43.724241 11122 ts_tablet_manager.cc:1397] T fa72812a7b4e4340a268b6e495be4cbe P 5a95da8804ee41d6a672ee947fab39df: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 19:01:43.724404 11120 raft_consensus.cc:348] T fa72812a7b4e4340a268b6e495be4cbe P ce010af0e43140678ac8b9cb7d252266 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce010af0e43140678ac8b9cb7d252266" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45037 } } peers { permanent_uuid: "5a95da8804ee41d6a672ee947fab39df" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36111 } } peers { permanent_uuid: "66d15106dd5746d6a1ab2074ad38fbff" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46739 } }
I20260929 19:01:43.724581 11120 raft_consensus.cc:374] T fa72812a7b4e4340a268b6e495be4cbe P ce010af0e43140678ac8b9cb7d252266 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:01:43.724633 11120 raft_consensus.cc:729] T fa72812a7b4e4340a268b6e495be4cbe P ce010af0e43140678ac8b9cb7d252266 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce010af0e43140678ac8b9cb7d252266, State: Initialized, Role: FOLLOWER
I20260929 19:01:43.724746 11120 consensus_queue.cc:260] T fa72812a7b4e4340a268b6e495be4cbe P ce010af0e43140678ac8b9cb7d252266 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce010af0e43140678ac8b9cb7d252266" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45037 } } peers { permanent_uuid: "5a95da8804ee41d6a672ee947fab39df" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36111 } } peers { permanent_uuid: "66d15106dd5746d6a1ab2074ad38fbff" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46739 } }
I20260929 19:01:43.725060 11120 ts_tablet_manager.cc:1428] T fa72812a7b4e4340a268b6e495be4cbe P ce010af0e43140678ac8b9cb7d252266: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 19:01:43.725087 10836 heartbeater.cc:503] Master 127.0.98.62:33845 was elected leader, sending a full tablet report...
I20260929 19:01:43.726270 11122 raft_consensus.cc:348] T fa72812a7b4e4340a268b6e495be4cbe P 5a95da8804ee41d6a672ee947fab39df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce010af0e43140678ac8b9cb7d252266" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45037 } } peers { permanent_uuid: "5a95da8804ee41d6a672ee947fab39df" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36111 } } peers { permanent_uuid: "66d15106dd5746d6a1ab2074ad38fbff" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46739 } }
I20260929 19:01:43.726428 11122 raft_consensus.cc:374] T fa72812a7b4e4340a268b6e495be4cbe P 5a95da8804ee41d6a672ee947fab39df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:01:43.726470 11122 raft_consensus.cc:729] T fa72812a7b4e4340a268b6e495be4cbe P 5a95da8804ee41d6a672ee947fab39df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a95da8804ee41d6a672ee947fab39df, State: Initialized, Role: FOLLOWER
I20260929 19:01:43.726567 11122 consensus_queue.cc:260] T fa72812a7b4e4340a268b6e495be4cbe P 5a95da8804ee41d6a672ee947fab39df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce010af0e43140678ac8b9cb7d252266" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45037 } } peers { permanent_uuid: "5a95da8804ee41d6a672ee947fab39df" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36111 } } peers { permanent_uuid: "66d15106dd5746d6a1ab2074ad38fbff" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46739 } }
I20260929 19:01:43.726801 11122 ts_tablet_manager.cc:1428] T fa72812a7b4e4340a268b6e495be4cbe P 5a95da8804ee41d6a672ee947fab39df: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 19:01:43.726852 11104 heartbeater.cc:503] Master 127.0.98.62:33845 was elected leader, sending a full tablet report...
W20260929 19:01:43.774392 10971 tablet.cc:2369] T fa72812a7b4e4340a268b6e495be4cbe P 66d15106dd5746d6a1ab2074ad38fbff: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 19:01:43.887636 10837 tablet.cc:2369] T fa72812a7b4e4340a268b6e495be4cbe P ce010af0e43140678ac8b9cb7d252266: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 19:01:43.916509 11125 raft_consensus.cc:482] T fa72812a7b4e4340a268b6e495be4cbe P ce010af0e43140678ac8b9cb7d252266 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 19:01:43.916651 11125 raft_consensus.cc:504] T fa72812a7b4e4340a268b6e495be4cbe P ce010af0e43140678ac8b9cb7d252266 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce010af0e43140678ac8b9cb7d252266" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45037 } } peers { permanent_uuid: "5a95da8804ee41d6a672ee947fab39df" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36111 } } peers { permanent_uuid: "66d15106dd5746d6a1ab2074ad38fbff" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46739 } }
I20260929 19:01:43.917033 11125 leader_election.cc:288] T fa72812a7b4e4340a268b6e495be4cbe P ce010af0e43140678ac8b9cb7d252266 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5a95da8804ee41d6a672ee947fab39df (127.0.98.3:36111), 66d15106dd5746d6a1ab2074ad38fbff (127.0.98.2:46739)
I20260929 19:01:43.919983 11116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:43.917289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58565 (local address 127.0.98.3:36111)
0929 19:01:43.917339 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:43.917342 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:43.917477 (+ 135us) server_negotiation.cc:407] Connection header received
0929 19:01:43.917534 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:43.917535 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:43.917548 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:43.917623 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:01:43.917818 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:43.918251 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:43.919262 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:43.919635 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:43.919720 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:43.919729 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0929 19:01:43.919874 (+ 145us) server_negotiation.cc:299] Negotiation successful
0929 19:01:43.919911 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260929 19:01:43.919994 11117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:43.917262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45861 (local address 127.0.98.2:46739)
0929 19:01:43.917306 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:43.917308 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:43.917457 (+ 149us) server_negotiation.cc:407] Connection header received
0929 19:01:43.917519 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:43.917520 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:43.917532 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:43.917589 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:01:43.917770 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:43.918246 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:43.919264 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:43.919635 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:43.919718 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:43.919729 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0929 19:01:43.919887 (+ 158us) server_negotiation.cc:299] Negotiation successful
0929 19:01:43.919927 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260929 19:01:43.920020 11128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:43.917196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36111 (local address 127.0.98.1:58565)
0929 19:01:43.917360 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:43.917369 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 19:01:43.917475 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:01:43.917640 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:01:43.917641 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:01:43.917643 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:01:43.917725 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:43.917726 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:43.918348 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:43.918349 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:43.919171 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:43.919173 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:43.919720 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:43.919721 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:43.919790 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:43.919794 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 19:01:43.919869 (+ 75us) client_negotiation.cc:240] Negotiation successful
0929 19:01:43.919921 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"mutex_wait_us":29,"thread_start_us":51,"threads_started":1}
I20260929 19:01:43.920022 11127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:43.917196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46739 (local address 127.0.98.1:45861)
0929 19:01:43.917327 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:43.917339 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 19:01:43.917437 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:01:43.917617 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:01:43.917618 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:01:43.917621 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:01:43.917707 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:43.917708 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:43.918330 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:43.918331 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:43.919173 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:01:43.919175 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:01:43.919722 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:01:43.919723 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:01:43.919792 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:43.919802 (+ 10us) client_negotiation.cc:769] Sending connection context
0929 19:01:43.919878 (+ 76us) client_negotiation.cc:240] Negotiation successful
0929 19:01:43.919924 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"mutex_wait_us":54,"thread_start_us":62,"threads_started":1}
I20260929 19:01:43.920354 11058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa72812a7b4e4340a268b6e495be4cbe" candidate_uuid: "ce010af0e43140678ac8b9cb7d252266" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a95da8804ee41d6a672ee947fab39df" is_pre_election: true
I20260929 19:01:43.920358 10924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa72812a7b4e4340a268b6e495be4cbe" candidate_uuid: "ce010af0e43140678ac8b9cb7d252266" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66d15106dd5746d6a1ab2074ad38fbff" is_pre_election: true
I20260929 19:01:43.920514 11058 raft_consensus.cc:2454] T fa72812a7b4e4340a268b6e495be4cbe P 5a95da8804ee41d6a672ee947fab39df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce010af0e43140678ac8b9cb7d252266 in term 0.
I20260929 19:01:43.920528 10924 raft_consensus.cc:2454] T fa72812a7b4e4340a268b6e495be4cbe P 66d15106dd5746d6a1ab2074ad38fbff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce010af0e43140678ac8b9cb7d252266 in term 0.
I20260929 19:01:43.920732 10726 leader_election.cc:302] T fa72812a7b4e4340a268b6e495be4cbe P ce010af0e43140678ac8b9cb7d252266 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a95da8804ee41d6a672ee947fab39df, ce010af0e43140678ac8b9cb7d252266; no voters:
I20260929 19:01:43.920892 11125 raft_consensus.cc:2781] T fa72812a7b4e4340a268b6e495be4cbe P ce010af0e43140678ac8b9cb7d252266 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 19:01:43.920953 11125 raft_consensus.cc:482] T fa72812a7b4e4340a268b6e495be4cbe P ce010af0e43140678ac8b9cb7d252266 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 19:01:43.920975 11125 raft_consensus.cc:3037] T fa72812a7b4e4340a268b6e495be4cbe P ce010af0e43140678ac8b9cb7d252266 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:01:43.921648 11125 raft_consensus.cc:504] T fa72812a7b4e4340a268b6e495be4cbe P ce010af0e43140678ac8b9cb7d252266 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce010af0e43140678ac8b9cb7d252266" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45037 } } peers { permanent_uuid: "5a95da8804ee41d6a672ee947fab39df" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36111 } } peers { permanent_uuid: "66d15106dd5746d6a1ab2074ad38fbff" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46739 } }
I20260929 19:01:43.921792 11125 leader_election.cc:288] T fa72812a7b4e4340a268b6e495be4cbe P ce010af0e43140678ac8b9cb7d252266 [CANDIDATE]: Term 1 election: Requested vote from peers 5a95da8804ee41d6a672ee947fab39df (127.0.98.3:36111), 66d15106dd5746d6a1ab2074ad38fbff (127.0.98.2:46739)
I20260929 19:01:43.921931 10924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa72812a7b4e4340a268b6e495be4cbe" candidate_uuid: "ce010af0e43140678ac8b9cb7d252266" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66d15106dd5746d6a1ab2074ad38fbff"
I20260929 19:01:43.921947 11058 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa72812a7b4e4340a268b6e495be4cbe" candidate_uuid: "ce010af0e43140678ac8b9cb7d252266" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a95da8804ee41d6a672ee947fab39df"
I20260929 19:01:43.922001 10924 raft_consensus.cc:3037] T fa72812a7b4e4340a268b6e495be4cbe P 66d15106dd5746d6a1ab2074ad38fbff [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:01:43.922030 11058 raft_consensus.cc:3037] T fa72812a7b4e4340a268b6e495be4cbe P 5a95da8804ee41d6a672ee947fab39df [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:01:43.922730 11058 raft_consensus.cc:2454] T fa72812a7b4e4340a268b6e495be4cbe P 5a95da8804ee41d6a672ee947fab39df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce010af0e43140678ac8b9cb7d252266 in term 1.
I20260929 19:01:43.922807 10924 raft_consensus.cc:2454] T fa72812a7b4e4340a268b6e495be4cbe P 66d15106dd5746d6a1ab2074ad38fbff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce010af0e43140678ac8b9cb7d252266 in term 1.
I20260929 19:01:43.922888 10726 leader_election.cc:302] T fa72812a7b4e4340a268b6e495be4cbe P ce010af0e43140678ac8b9cb7d252266 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a95da8804ee41d6a672ee947fab39df, ce010af0e43140678ac8b9cb7d252266; no voters:
I20260929 19:01:43.922966 11125 raft_consensus.cc:2781] T fa72812a7b4e4340a268b6e495be4cbe P ce010af0e43140678ac8b9cb7d252266 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:01:43.923116 11125 raft_consensus.cc:686] T fa72812a7b4e4340a268b6e495be4cbe P ce010af0e43140678ac8b9cb7d252266 [term 1 LEADER]: Becoming Leader. State: Replica: ce010af0e43140678ac8b9cb7d252266, State: Running, Role: LEADER
I20260929 19:01:43.923228 11125 consensus_queue.cc:237] T fa72812a7b4e4340a268b6e495be4cbe P ce010af0e43140678ac8b9cb7d252266 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce010af0e43140678ac8b9cb7d252266" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45037 } } peers { permanent_uuid: "5a95da8804ee41d6a672ee947fab39df" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36111 } } peers { permanent_uuid: "66d15106dd5746d6a1ab2074ad38fbff" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46739 } }
I20260929 19:01:43.924077 10644 catalog_manager.cc:5495] T fa72812a7b4e4340a268b6e495be4cbe P ce010af0e43140678ac8b9cb7d252266 reported cstate change: term changed from 0 to 1, leader changed from <none> to ce010af0e43140678ac8b9cb7d252266 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "ce010af0e43140678ac8b9cb7d252266" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce010af0e43140678ac8b9cb7d252266" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5a95da8804ee41d6a672ee947fab39df" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36111 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66d15106dd5746d6a1ab2074ad38fbff" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46739 } health_report { overall_health: UNKNOWN } } }
W20260929 19:01:43.929527 11105 tablet.cc:2369] T fa72812a7b4e4340a268b6e495be4cbe P 5a95da8804ee41d6a672ee947fab39df: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 19:01:43.938153 11115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:43.936064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58822 (local address 127.0.98.1:45037)
0929 19:01:43.936112 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0929 19:01:43.936114 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:01:43.936120 (+ 6us) server_negotiation.cc:407] Connection header received
0929 19:01:43.936144 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:01:43.936145 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:01:43.936159 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:01:43.936226 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 19:01:43.936348 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:43.936778 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:43.937599 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:01:43.937762 (+ 163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:01:43.937806 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:01:43.937929 (+ 123us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 19:01:43.938009 (+ 80us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 19:01:43.938065 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0929 19:01:43.938096 (+ 31us) server_negotiation.cc:299] Negotiation successful
0929 19:01:43.938109 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260929 19:01:43.939663 11058 raft_consensus.cc:1261] T fa72812a7b4e4340a268b6e495be4cbe P 5a95da8804ee41d6a672ee947fab39df [term 1 FOLLOWER]: Refusing update from remote peer ce010af0e43140678ac8b9cb7d252266: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 19:01:43.939667 10924 raft_consensus.cc:1261] T fa72812a7b4e4340a268b6e495be4cbe P 66d15106dd5746d6a1ab2074ad38fbff [term 1 FOLLOWER]: Refusing update from remote peer ce010af0e43140678ac8b9cb7d252266: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 19:01:43.939945 11125 consensus_queue.cc:1035] T fa72812a7b4e4340a268b6e495be4cbe P ce010af0e43140678ac8b9cb7d252266 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a95da8804ee41d6a672ee947fab39df" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 19:01:43.940102 11129 consensus_queue.cc:1035] T fa72812a7b4e4340a268b6e495be4cbe P ce010af0e43140678ac8b9cb7d252266 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66d15106dd5746d6a1ab2074ad38fbff" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46739 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 19:01:43.942394 11133 mvcc.cc:205] Tried to move back new op lower bound from 7334742032134574080 to 7334742032070180864. Current Snapshot: MvccSnapshot[applied={T|T < 7334742032134574080}]
I20260929 19:01:43.944243 10645 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37880:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 19:01:43.944329 10645 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37880:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 19:01:43.945037 10645 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b0c1f78bc00e4da0a1d58b6ab7878157: Sending DeleteTablet for 3 replicas of tablet fa72812a7b4e4340a268b6e495be4cbe
I20260929 19:01:43.945333 10770 tablet_service.cc:1514] Processing DeleteTablet for tablet fa72812a7b4e4340a268b6e495be4cbe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:01:43 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58812
I20260929 19:01:43.945353 10904 tablet_service.cc:1514] Processing DeleteTablet for tablet fa72812a7b4e4340a268b6e495be4cbe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:01:43 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50024
I20260929 19:01:43.945379 11038 tablet_service.cc:1514] Processing DeleteTablet for tablet fa72812a7b4e4340a268b6e495be4cbe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:01:43 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40240
I20260929 19:01:43.945568 11142 tablet_replica.cc:330] T fa72812a7b4e4340a268b6e495be4cbe P 5a95da8804ee41d6a672ee947fab39df: stopping tablet replica
I20260929 19:01:43.945574 11143 tablet_replica.cc:330] T fa72812a7b4e4340a268b6e495be4cbe P 66d15106dd5746d6a1ab2074ad38fbff: stopping tablet replica
I20260929 19:01:43.945657 11143 raft_consensus.cc:2229] T fa72812a7b4e4340a268b6e495be4cbe P 66d15106dd5746d6a1ab2074ad38fbff [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 19:01:43.945657 11142 raft_consensus.cc:2229] T fa72812a7b4e4340a268b6e495be4cbe P 5a95da8804ee41d6a672ee947fab39df [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 19:01:43.945749 11143 raft_consensus.cc:2258] T fa72812a7b4e4340a268b6e495be4cbe P 66d15106dd5746d6a1ab2074ad38fbff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 19:01:43.945749 11142 raft_consensus.cc:2258] T fa72812a7b4e4340a268b6e495be4cbe P 5a95da8804ee41d6a672ee947fab39df [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 19:01:43.945884 11141 tablet_replica.cc:330] T fa72812a7b4e4340a268b6e495be4cbe P ce010af0e43140678ac8b9cb7d252266: stopping tablet replica
I20260929 19:01:43.946025 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUbnLHE/build/release/bin/kudu with pid 10707
I20260929 19:01:43.946203 11143 ts_tablet_manager.cc:1905] T fa72812a7b4e4340a268b6e495be4cbe P 66d15106dd5746d6a1ab2074ad38fbff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 19:01:43.946224 11142 ts_tablet_manager.cc:1905] T fa72812a7b4e4340a268b6e495be4cbe P 5a95da8804ee41d6a672ee947fab39df: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 19:01:43.947855 11143 ts_tablet_manager.cc:1918] T fa72812a7b4e4340a268b6e495be4cbe P 66d15106dd5746d6a1ab2074ad38fbff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 19:01:43.947855 11142 ts_tablet_manager.cc:1918] T fa72812a7b4e4340a268b6e495be4cbe P 5a95da8804ee41d6a672ee947fab39df: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 19:01:43.947937 11143 log.cc:1196] T fa72812a7b4e4340a268b6e495be4cbe P 66d15106dd5746d6a1ab2074ad38fbff: Deleting WAL directory at /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708447806542-392-0/minicluster-data/ts-1/wal/wals/fa72812a7b4e4340a268b6e495be4cbe
I20260929 19:01:43.947943 11142 log.cc:1196] T fa72812a7b4e4340a268b6e495be4cbe P 5a95da8804ee41d6a672ee947fab39df: Deleting WAL directory at /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790708447806542-392-0/minicluster-data/ts-2/wal/wals/fa72812a7b4e4340a268b6e495be4cbe
I20260929 19:01:43.948225 11143 ts_tablet_manager.cc:1939] T fa72812a7b4e4340a268b6e495be4cbe P 66d15106dd5746d6a1ab2074ad38fbff: Deleting consensus metadata
I20260929 19:01:43.948225 11142 ts_tablet_manager.cc:1939] T fa72812a7b4e4340a268b6e495be4cbe P 5a95da8804ee41d6a672ee947fab39df: Deleting consensus metadata
I20260929 19:01:43.948559 10632 catalog_manager.cc:4841] TS 5a95da8804ee41d6a672ee947fab39df (127.0.98.3:36111): tablet fa72812a7b4e4340a268b6e495be4cbe (table test-table [id=9dd00c2179b74ae19a27ce9859f0dddd]) successfully deleted
I20260929 19:01:43.948714 10629 catalog_manager.cc:4841] TS 66d15106dd5746d6a1ab2074ad38fbff (127.0.98.2:46739): tablet fa72812a7b4e4340a268b6e495be4cbe (table test-table [id=9dd00c2179b74ae19a27ce9859f0dddd]) successfully deleted
W20260929 19:01:43.951625 10632 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:45037 (error 108)
I20260929 19:01:43.951845 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUbnLHE/build/release/bin/kudu with pid 10841
I20260929 19:01:43.952040 11112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:01:43.951941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45037 (local address 127.0.0.1:58832)
0929 19:01:43.951982 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0929 19:01:43.952008 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45037: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260929 19:01:43.952378 10632 catalog_manager.cc:4584] TS ce010af0e43140678ac8b9cb7d252266 (127.0.98.1:45037): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet fa72812a7b4e4340a268b6e495be4cbe: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45037: connect: Connection refused (error 111)
I20260929 19:01:43.956518 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUbnLHE/build/release/bin/kudu with pid 10975
I20260929 19:01:43.961716 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUbnLHE/build/release/bin/kudu with pid 10613
2026-09-29T19:01:43Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2018 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708447806542-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 19:01:44 dist-test-slave-lqkv krb5kdc[11147](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 19:01:44 dist-test-slave-lqkv krb5kdc[11147](info): set up 2 sockets
Sep 29 19:01:44 dist-test-slave-lqkv krb5kdc[11147](info): commencing operation
krb5kdc: starting...
W20260929 19:01:44.981477 392 mini_kdc.cc:121] Time spent starting KDC: real 1.001s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708447806542-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708447806542-392-0/krb5kdc/test-user.keytab.
Sep 29 19:01:45 dist-test-slave-lqkv krb5kdc[11147](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708505, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T19:01:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T19:01:45Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708447806542-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708447806542-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708447806542-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708447806542-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708447806542-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708447806542-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260929 19:01:45.124598 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708447806542-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskUbnLHE/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708447806542-392-0/minicluster-data/postgres -l logfile start
2026-09-29 19:01:47.367 UTC [11177] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-29 19:01:47.367 UTC [11177] LOG: listening on IPv4 address "127.0.98.20", port 44695
2026-09-29 19:01:47.370 UTC [11177] LOG: listening on Unix socket "/tmp/.s.PGSQL.44695"
2026-09-29 19:01:47.403 UTC [11180] LOG: database system was shut down at 2026-09-29 19:01:46 UTC
2026-09-29 19:01:47.407 UTC [11177] LOG: database system is ready to accept connections
I20260929 19:01:48.329486 392 mini_postgres.cc:96] Postgres bound to 44695
2026-09-29 19:01:48.335 UTC [11188] FATAL: database "slave" does not exist
127.0.98.20:44695 - accepting connections
I20260929 19:01:48.335734 392 mini_ranger.cc:161] Starting Ranger...
I20260929 19:01:48.349485 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260929 19:01:48.749514 392 mini_ranger.cc:87] Created ranger Postgres database
I20260929 19:01:48.749579 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708447806542-392-0/minicluster-data/ranger-admin
2026-09-29 19:01:49,013 [I] DB FLAVOR :POSTGRES
2026-09-29 19:01:49,013 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 19:01:49,013 [I] Checking connection..
2026-09-29 19:01:49,013 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUbnLHE/build/release/bin/postgresql.jar:/tmp/dist-test-taskUbnLHE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:44695/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-29 19:01:49,263 [I] Checking connection passed.
2026-09-29 19:01:49,263 [I] --------- Verifying version history table ---------
2026-09-29 19:01:49,263 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUbnLHE/build/release/bin/postgresql.jar:/tmp/dist-test-taskUbnLHE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:44695/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 19:01:49,512 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 19:01:49,512 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUbnLHE/build/release/bin/postgresql.jar:/tmp/dist-test-taskUbnLHE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:44695/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 19:01:49,760 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 19:01:49,760 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-29 19:01:49,761 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUbnLHE/build/release/bin/postgresql.jar:/tmp/dist-test-taskUbnLHE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:44695/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskUbnLHE/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-29 19:01:49.981 UTC [11252] WARNING: there is no transaction in progress
2026-09-29 19:01:49,987 [I] create_dbversion_catalog.sql file imported successfully
2026-09-29 19:01:49,987 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUbnLHE/build/release/bin/postgresql.jar:/tmp/dist-test-taskUbnLHE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:44695/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 19:01:50,230 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-29 19:01:50,230 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 19:01:50,231 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUbnLHE/build/release/bin/postgresql.jar:/tmp/dist-test-taskUbnLHE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:44695/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-29 19:01:50,472 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUbnLHE/build/release/bin/postgresql.jar:/tmp/dist-test-taskUbnLHE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:44695/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-29 19:01:50,710 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUbnLHE/build/release/bin/postgresql.jar:/tmp/dist-test-taskUbnLHE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:44695/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-lqkv.c.gcp-upstream.internal','N') ;"
2026-09-29 19:01:50,942 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUbnLHE/build/release/bin/postgresql.jar:/tmp/dist-test-taskUbnLHE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:44695/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 19:01:51,173 [I] Table x_portal_user does not exist in database ranger
2026-09-29 19:01:51,173 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUbnLHE/build/release/bin/postgresql.jar:/tmp/dist-test-taskUbnLHE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:44695/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 19:01:51,413 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-29 19:01:51,413 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUbnLHE/build/release/bin/postgresql.jar:/tmp/dist-test-taskUbnLHE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:44695/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 19:01:51,650 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUbnLHE/build/release/bin/postgresql.jar:/tmp/dist-test-taskUbnLHE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:44695/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 19:01:51,885 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-29 19:01:51,886 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUbnLHE/build/release/bin/postgresql.jar:/tmp/dist-test-taskUbnLHE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:44695/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskUbnLHE/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-29 19:01:52.769 UTC [11360] WARNING: there is no transaction in progress
2026-09-29 19:01:52.784 UTC [11360] WARNING: there is no transaction in progress
2026-09-29 19:01:52.799 UTC [11360] WARNING: there is no transaction in progress
2026-09-29 19:01:52.813 UTC [11360] WARNING: there is no transaction in progress
2026-09-29 19:01:52.829 UTC [11360] WARNING: there is no transaction in progress
2026-09-29 19:01:52.845 UTC [11360] WARNING: there is no transaction in progress
2026-09-29 19:01:52.904 UTC [11360] WARNING: there is no transaction in progress
2026-09-29 19:01:52.910 UTC [11360] WARNING: there is no transaction in progress
2026-09-29 19:01:52.923 UTC [11360] WARNING: there is no transaction in progress
2026-09-29 19:01:52.932 UTC [11360] WARNING: there is no transaction in progress
2026-09-29 19:01:52.941 UTC [11360] WARNING: there is no transaction in progress
2026-09-29 19:01:52.953 UTC [11360] WARNING: there is no transaction in progress
2026-09-29 19:01:52.961 UTC [11360] WARNING: there is no transaction in progress
2026-09-29 19:01:52.967 UTC [11360] WARNING: there is no transaction in progress
2026-09-29 19:01:52.972 UTC [11360] WARNING: there is no transaction in progress
2026-09-29 19:01:52.977 UTC [11360] WARNING: there is no transaction in progress
2026-09-29 19:01:53,524 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-29 19:01:53,524 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUbnLHE/build/release/bin/postgresql.jar:/tmp/dist-test-taskUbnLHE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:44695/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-29 19:01:53,774 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-29 19:01:53,775 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUbnLHE/build/release/bin/postgresql.jar:/tmp/dist-test-taskUbnLHE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:44695/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 19:01:54,019 [I] Table x_portal_user already exists in database 'ranger'
2026-09-29 19:01:54,019 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUbnLHE/build/release/bin/postgresql.jar:/tmp/dist-test-taskUbnLHE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:44695/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 19:01:54,230 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-29 19:01:54,231 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUbnLHE/build/release/bin/postgresql.jar:/tmp/dist-test-taskUbnLHE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:44695/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 19:01:54,463 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUbnLHE/build/release/bin/postgresql.jar:/tmp/dist-test-taskUbnLHE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:44695/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 19:01:54,679 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUbnLHE/build/release/bin/postgresql.jar:/tmp/dist-test-taskUbnLHE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:44695/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-lqkv.c.gcp-upstream.internal';"
2026-09-29 19:01:54,913 [I] CORE_DB_SCHEMA import status has been updated
2026-09-29 19:01:54,914 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUbnLHE/build/release/bin/postgresql.jar:/tmp/dist-test-taskUbnLHE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:44695/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-29 19:01:55,126 [I] DB_PATCHES have already been applied
I20260929 19:01:55.131675 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708447806542-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskUbnLHE/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskUbnLHE/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskUbnLHE/build/release/bin/postgresql.jar:/tmp/dist-test-taskUbnLHE/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260929 19:01:55.131731 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260929 19:01:55.134459 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:55299
Sep 29, 2026 7:01:55 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 7:01:55 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 7:01:55 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708447806542-392-0/minicluster-data/ranger-admin/ews/webapp
Sep 29, 2026 7:01:55 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708447806542-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 29, 2026 7:01:55 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708447806542-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 29, 2026 7:01:55 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708447806542-392-0/minicluster-data/ranger-admin/ews/webapp].
Sep 29, 2026 7:01:55 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260929 19:01:56.244838 392 mini_ranger.cc:160] Time spent starting Ranger: real 7.909s user 0.000s sys 0.008s
Sep 29, 2026 7:02:14 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 29, 2026 7:02:14 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.MetricsREST
Sep 29, 2026 7:02:14 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 29, 2026 7:02:14 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 29, 2026 7:02:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 29, 2026 7:02:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 29, 2026 7:02:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 29, 2026 7:02:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 29, 2026 7:02:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 29, 2026 7:02:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 29, 2026 7:02:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 29, 2026 7:02:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 29, 2026 7:02:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 29, 2026 7:02:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 29, 2026 7:02:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 29, 2026 7:02:14 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 29, 2026 7:02:14 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 29, 2026 7:02:15 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260929 19:02:15.775696 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708447806542-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708447806542-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708447806542-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708447806542-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 29 19:02:15 dist-test-slave-lqkv krb5kdc[11147](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708535, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260929 19:02:15.846944 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260929 19:02:15.854545 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260929 19:02:16.589102 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260929 19:02:16.589182 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708447806542-392-0/minicluster-data/ranger-kms
I20260929 19:02:16.589200 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:44695
2026-09-29 19:02:16,758 [I] DB FLAVOR :POSTGRES
2026-09-29 19:02:16,758 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 19:02:16,758 [I] Checking connection
2026-09-29 19:02:16,758 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUbnLHE/build/release/bin/postgresql.jar:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708447806542-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:44695/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-29 19:02:17,008 [I] connection success
2026-09-29 19:02:17,008 [I] --------- Verifying Ranger DB tables ---------
2026-09-29 19:02:17,008 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-29 19:02:17,008 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUbnLHE/build/release/bin/postgresql.jar:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708447806542-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:44695/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-29 19:02:17,224 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-29 19:02:17,224 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 19:02:17,224 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-29 19:02:17,225 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUbnLHE/build/release/bin/postgresql.jar:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708447806542-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:44695/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708447806542-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-29 19:02:17,464 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260929 19:02:17.907269 392 mini_ranger_kms.cc:310] Created kms service
I20260929 19:02:18.084714 392 mini_ranger_kms.cc:326] Created kudu user
I20260929 19:02:18.124378 392 mini_ranger_kms.cc:343] Created rangerkms user
Sep 29 19:02:18 dist-test-slave-lqkv krb5kdc[11147](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708535, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260929 19:02:18.496162 392 mini_ranger_kms.cc:384] Added ranger policy
I20260929 19:02:18.496269 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708447806542-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskUbnLHE/build/release/bin/postgresql.jar:/tmp/dist-test-taskUbnLHE/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskUbnLHE/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260929 19:02:18.496289 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260929 19:02:18.499541 392 mini_ranger_kms.cc:276] Ranger KMS PID: 11540
I20260929 19:02:18.499658 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:52183
Sep 29, 2026 7:02:19 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 7:02:19 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 7:02:19 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708447806542-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 29, 2026 7:02:19 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708447806542-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 29, 2026 7:02:20 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708447806542-392-0/minicluster-data/ranger-kms/ews/webapp].
Sep 29, 2026 7:02:20 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260929 19:02:20.641383 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.794s user 0.006s sys 0.146s
I20260929 19:02:20.641499 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260929 19:02:20.641585 392 mini_ranger_kms.cc:401] 127.0.98.20:52183/kms/v1/keys
Sep 29 19:02:20 dist-test-slave-lqkv krb5kdc[11147](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790708540, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 29 19:02:21 dist-test-slave-lqkv krb5kdc[11147](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708540, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 29 19:02:21 dist-test-slave-lqkv krb5kdc[11147](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708540, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 29 19:02:23 dist-test-slave-lqkv krb5kdc[11147](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708543, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708447806542-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708447806542-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 19:02:23.450968 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUbnLHE/build/release/bin/kudu
/tmp/dist-test-taskUbnLHE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708447806542-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708447806542-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708447806542-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708447806542-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33605
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708447806542-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33605
--ranger_config_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708447806542-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:52183/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708447806542-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708447806542-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708447806542-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:02:23.552450 11591 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:02:23.552634 11591 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:02:23.552654 11591 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:02:23.553742 11591 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 19:02:23.553781 11591 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:02:23.553793 11591 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:02:23.553804 11591 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 19:02:23.553815 11591 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 19:02:23.555751 11591 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708447806542-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708447806542-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:52183/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33605
--ranger_config_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708447806542-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708447806542-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33605
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708447806542-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11591
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708447806542-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 18:09:13 UTC on ccycloud.quasar-hpkalf.root.comops.site
build id 4680913
I20260929 19:02:23.555963 11591 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:02:23.556191 11591 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:02:23.561487 11596 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:02:23.561564 11597 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:02:23.561782 11591 server_base.cc:1034] running on GCE node
W20260929 19:02:23.561780 11599 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:02:23.562109 11591 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:02:23.562394 11591 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:02:23.563576 11591 hybrid_clock.cc:648] HybridClock initialized: now 1790708543563552 us; error 37 us; skew 500 ppm
Sep 29 19:02:23 dist-test-slave-lqkv krb5kdc[11147](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708543, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:02:23.566378 11591 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 19:02:23.566829 11591 webserver.cc:460] Webserver started at http://127.0.98.62:41739/ using document root <none> and password file <none>
I20260929 19:02:23.567031 11591 fs_manager.cc:359] Metadata directory not provided
I20260929 19:02:23.567082 11591 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:02:23.567205 11591 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 19:02:23 dist-test-slave-lqkv krb5kdc[11147](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708543, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260929 19:02:23.616230 11591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708447806542-392-0/minicluster-data/master-0/data/instance:
uuid: "b1cbc4fa8c374a72ab0043f6d3b417e1"
format_stamp: "Formatted at 2026-09-29 19:02:23 on dist-test-slave-lqkv"
server_key: "598c97cb97bbefc8b939d8fd2e5dec08"
server_key_iv: "c2c42f82f008884aeb1565ee319281c0"
server_key_version: "kuduclusterkey@0"
I20260929 19:02:23.616662 11591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708447806542-392-0/minicluster-data/master-0/wal/instance:
uuid: "b1cbc4fa8c374a72ab0043f6d3b417e1"
format_stamp: "Formatted at 2026-09-29 19:02:23 on dist-test-slave-lqkv"
server_key: "598c97cb97bbefc8b939d8fd2e5dec08"
server_key_iv: "c2c42f82f008884aeb1565ee319281c0"
server_key_version: "kuduclusterkey@0"
I20260929 19:02:23.618451 11591 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.001s
I20260929 19:02:23.631435 11606 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:02:23.631660 11591 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 19:02:23.631762 11591 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708447806542-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708447806542-392-0/minicluster-data/master-0/wal
uuid: "b1cbc4fa8c374a72ab0043f6d3b417e1"
format_stamp: "Formatted at 2026-09-29 19:02:23 on dist-test-slave-lqkv"
server_key: "598c97cb97bbefc8b939d8fd2e5dec08"
server_key_iv: "c2c42f82f008884aeb1565ee319281c0"
server_key_version: "kuduclusterkey@0"
I20260929 19:02:23.631843 11591 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708447806542-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708447806542-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708447806542-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:02:23.642750 11591 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:02:23.644438 11591 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:02:23.644584 11591 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:02:23.649309 11591 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33605
I20260929 19:02:23.649328 11658 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33605 every 8 connection(s)
I20260929 19:02:23.649658 11591 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708447806542-392-0/minicluster-data/master-0/data/info.pb
I20260929 19:02:23.650270 11659 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:02:23.652232 11659 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b1cbc4fa8c374a72ab0043f6d3b417e1: Bootstrap starting.
I20260929 19:02:23.652750 11659 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b1cbc4fa8c374a72ab0043f6d3b417e1: Neither blocks nor log segments found. Creating new log.
I20260929 19:02:23.653020 11659 log.cc:824] T 00000000000000000000000000000000 P b1cbc4fa8c374a72ab0043f6d3b417e1: Log is configured to *not* fsync() on all Append() calls
I20260929 19:02:23.653630 11659 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b1cbc4fa8c374a72ab0043f6d3b417e1: No bootstrap required, opened a new log
I20260929 19:02:23.655941 11659 raft_consensus.cc:348] T 00000000000000000000000000000000 P b1cbc4fa8c374a72ab0043f6d3b417e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1cbc4fa8c374a72ab0043f6d3b417e1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33605 } }
I20260929 19:02:23.656059 11659 raft_consensus.cc:374] T 00000000000000000000000000000000 P b1cbc4fa8c374a72ab0043f6d3b417e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:02:23.656086 11659 raft_consensus.cc:729] T 00000000000000000000000000000000 P b1cbc4fa8c374a72ab0043f6d3b417e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1cbc4fa8c374a72ab0043f6d3b417e1, State: Initialized, Role: FOLLOWER
I20260929 19:02:23.656229 11659 consensus_queue.cc:260] T 00000000000000000000000000000000 P b1cbc4fa8c374a72ab0043f6d3b417e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1cbc4fa8c374a72ab0043f6d3b417e1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33605 } }
I20260929 19:02:23.656342 11659 raft_consensus.cc:388] T 00000000000000000000000000000000 P b1cbc4fa8c374a72ab0043f6d3b417e1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 19:02:23.656414 11659 raft_consensus.cc:482] T 00000000000000000000000000000000 P b1cbc4fa8c374a72ab0043f6d3b417e1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 19:02:23.656471 11659 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b1cbc4fa8c374a72ab0043f6d3b417e1 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:02:23.656471 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUbnLHE/build/release/bin/kudu as pid 11591
I20260929 19:02:23.656553 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708447806542-392-0/minicluster-data/master-0/wal/instance
I20260929 19:02:23.656764 392 external_mini_cluster.cc:1428] Setting key 73a6bde1bd91c5e29313f2d70477c622
I20260929 19:02:23.657177 11659 raft_consensus.cc:504] T 00000000000000000000000000000000 P b1cbc4fa8c374a72ab0043f6d3b417e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1cbc4fa8c374a72ab0043f6d3b417e1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33605 } }
I20260929 19:02:23.657305 11659 leader_election.cc:302] T 00000000000000000000000000000000 P b1cbc4fa8c374a72ab0043f6d3b417e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b1cbc4fa8c374a72ab0043f6d3b417e1; no voters:
I20260929 19:02:23.657514 11659 leader_election.cc:288] T 00000000000000000000000000000000 P b1cbc4fa8c374a72ab0043f6d3b417e1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 19:02:23.657542 11664 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b1cbc4fa8c374a72ab0043f6d3b417e1 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:02:23.657831 11659 sys_catalog.cc:564] T 00000000000000000000000000000000 P b1cbc4fa8c374a72ab0043f6d3b417e1 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 19:02:23.658409 11664 raft_consensus.cc:686] T 00000000000000000000000000000000 P b1cbc4fa8c374a72ab0043f6d3b417e1 [term 1 LEADER]: Becoming Leader. State: Replica: b1cbc4fa8c374a72ab0043f6d3b417e1, State: Running, Role: LEADER
I20260929 19:02:23.659322 11664 consensus_queue.cc:237] T 00000000000000000000000000000000 P b1cbc4fa8c374a72ab0043f6d3b417e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1cbc4fa8c374a72ab0043f6d3b417e1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33605 } }
I20260929 19:02:23.659919 11665 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1cbc4fa8c374a72ab0043f6d3b417e1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b1cbc4fa8c374a72ab0043f6d3b417e1. Latest consensus state: current_term: 1 leader_uuid: "b1cbc4fa8c374a72ab0043f6d3b417e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1cbc4fa8c374a72ab0043f6d3b417e1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33605 } } }
I20260929 19:02:23.659981 11665 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1cbc4fa8c374a72ab0043f6d3b417e1 [sys.catalog]: This master's current role is: LEADER
I20260929 19:02:23.660166 11666 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1cbc4fa8c374a72ab0043f6d3b417e1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b1cbc4fa8c374a72ab0043f6d3b417e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1cbc4fa8c374a72ab0043f6d3b417e1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33605 } } }
I20260929 19:02:23.660233 11666 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1cbc4fa8c374a72ab0043f6d3b417e1 [sys.catalog]: This master's current role is: LEADER
I20260929 19:02:23.660392 11678 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 19:02:23.660606 11659 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708447806542-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260929 19:02:23.660912 11678 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 29 19:02:23 dist-test-slave-lqkv krb5kdc[11147](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708543, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 19:02:23.668825 11678 catalog_manager.cc:1347] Generated new cluster ID: 6b76803b26e04a049f2603f861627ee1
I20260929 19:02:23.668982 11678 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 19:02:23.671639 11663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:23.657252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40692 (local address 127.0.98.62:33605)
0929 19:02:23.657420 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:23.657429 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:23.657498 (+ 69us) server_negotiation.cc:407] Connection header received
0929 19:02:23.657621 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:23.657629 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:23.657689 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:23.657837 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:02:23.658612 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:23.660076 (+ 1464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:23.661220 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:23.664540 (+ 3320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:23.664705 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:23.667477 (+ 2772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:02:23.667481 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:02:23.667481 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:02:23.667492 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:02:23.670031 (+ 2539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:02:23.670407 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:02:23.670410 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:02:23.670410 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:02:23.670499 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:02:23.670698 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:02:23.670699 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:02:23.670699 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:02:23.671000 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:02:23.671149 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:23.671257 (+ 108us) server_negotiation.cc:299] Negotiation successful
0929 19:02:23.671396 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":72,"threads_started":1}
I20260929 19:02:23.680405 11678 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 19:02:23.681303 11678 catalog_manager.cc:1504] Loading token signing keys...
I20260929 19:02:23.688673 11678 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b1cbc4fa8c374a72ab0043f6d3b417e1: Generated new TSK 0
I20260929 19:02:23.688884 11678 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708447806542-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708447806542-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260929 19:02:24.353929 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUbnLHE/build/release/bin/kudu
/tmp/dist-test-taskUbnLHE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708447806542-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708447806542-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708447806542-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708447806542-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:52183/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33605
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708447806542-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708447806542-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708447806542-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708447806542-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:02:24.438453 11704 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:02:24.438666 11704 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:02:24.438725 11704 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:02:24.439857 11704 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:02:24.439936 11704 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:02:24.439978 11704 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 19:02:24.441442 11704 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708447806542-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708447806542-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:52183/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708447806542-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708447806542-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33605
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11704
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708447806542-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 18:09:13 UTC on ccycloud.quasar-hpkalf.root.comops.site
build id 4680913
I20260929 19:02:24.441680 11704 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:02:24.441896 11704 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:02:24.446552 11710 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:02:24.446712 11709 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:02:24.446555 11712 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:02:24.447237 11704 server_base.cc:1034] running on GCE node
I20260929 19:02:24.447417 11704 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:02:24.447659 11704 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:02:24.448861 11704 hybrid_clock.cc:648] HybridClock initialized: now 1790708544448837 us; error 37 us; skew 500 ppm
Sep 29 19:02:24 dist-test-slave-lqkv krb5kdc[11147](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708544, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:02:24.451190 11704 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 19:02:24.451566 11704 webserver.cc:460] Webserver started at http://127.0.98.1:41227/ using document root <none> and password file <none>
I20260929 19:02:24.451795 11704 fs_manager.cc:359] Metadata directory not provided
I20260929 19:02:24.451857 11704 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:02:24.451985 11704 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 19:02:24 dist-test-slave-lqkv krb5kdc[11147](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708544, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260929 19:02:24.463179 11704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708447806542-392-0/minicluster-data/ts-0/data/instance:
uuid: "f14b54af233143cf85bbff7c9bb1dbc8"
format_stamp: "Formatted at 2026-09-29 19:02:24 on dist-test-slave-lqkv"
server_key: "6d0710b4f2dd68e2bad95f5d327fffea"
server_key_iv: "53c7f88fa5cda8cfc4902efa391ed3cf"
server_key_version: "kuduclusterkey@0"
I20260929 19:02:24.463577 11704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708447806542-392-0/minicluster-data/ts-0/wal/instance:
uuid: "f14b54af233143cf85bbff7c9bb1dbc8"
format_stamp: "Formatted at 2026-09-29 19:02:24 on dist-test-slave-lqkv"
server_key: "6d0710b4f2dd68e2bad95f5d327fffea"
server_key_iv: "53c7f88fa5cda8cfc4902efa391ed3cf"
server_key_version: "kuduclusterkey@0"
I20260929 19:02:24.470619 11704 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.000s sys 0.002s
I20260929 19:02:24.488492 11719 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:02:24.488667 11704 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.000s sys 0.000s
I20260929 19:02:24.488740 11704 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708447806542-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708447806542-392-0/minicluster-data/ts-0/wal
uuid: "f14b54af233143cf85bbff7c9bb1dbc8"
format_stamp: "Formatted at 2026-09-29 19:02:24 on dist-test-slave-lqkv"
server_key: "6d0710b4f2dd68e2bad95f5d327fffea"
server_key_iv: "53c7f88fa5cda8cfc4902efa391ed3cf"
server_key_version: "kuduclusterkey@0"
I20260929 19:02:24.488826 11704 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708447806542-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708447806542-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708447806542-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:02:24.537887 11704 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:02:24.540287 11704 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:02:24.540445 11704 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:02:24.540663 11704 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:02:24.541052 11704 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:02:24.541106 11704 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:02:24.541145 11704 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:02:24.541172 11704 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:02:24.548784 11704 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45205
I20260929 19:02:24.548879 11834 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45205 every 8 connection(s)
I20260929 19:02:24.549193 11704 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708447806542-392-0/minicluster-data/ts-0/data/info.pb
I20260929 19:02:24.549448 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUbnLHE/build/release/bin/kudu as pid 11704
I20260929 19:02:24.549562 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708447806542-392-0/minicluster-data/ts-0/wal/instance
I20260929 19:02:24.549798 392 external_mini_cluster.cc:1428] Setting key 472d3a9ed8f742c890f375771855d5c0
Sep 29 19:02:24 dist-test-slave-lqkv krb5kdc[11147](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708544, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 19:02:24.559018 11838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:24.550691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55775 (local address 127.0.98.62:33605)
0929 19:02:24.550926 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:24.550928 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:24.550936 (+ 8us) server_negotiation.cc:407] Connection header received
0929 19:02:24.551160 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:24.551161 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:24.551174 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:24.551236 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:02:24.551779 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:24.552293 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:24.553595 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:24.553791 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:24.553862 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:24.555817 (+ 1955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:02:24.555819 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:02:24.555819 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:02:24.555833 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:02:24.557455 (+ 1622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:02:24.557950 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:02:24.557951 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:02:24.557952 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:02:24.558029 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:02:24.558322 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:02:24.558323 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:02:24.558323 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:02:24.558458 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:02:24.558513 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:24.558881 (+ 368us) server_negotiation.cc:299] Negotiation successful
0929 19:02:24.558947 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":211,"thread_start_us":102,"threads_started":1}
I20260929 19:02:24.559366 11837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:24.550120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33605 (local address 127.0.98.1:55775)
0929 19:02:24.550288 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0929 19:02:24.550329 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 19:02:24.551054 (+ 725us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:02:24.551310 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:02:24.551314 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:02:24.551492 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:02:24.551684 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:24.551687 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:24.552402 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:24.552402 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:24.553501 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:24.553503 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:24.553867 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:24.553868 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:24.553995 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:24.554456 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:02:24.554464 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:02:24.555725 (+ 1261us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:02:24.557564 (+ 1839us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:02:24.557570 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:02:24.557572 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:02:24.557878 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:02:24.558106 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:02:24.558107 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:02:24.558107 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:02:24.558252 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:02:24.558528 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:02:24.558533 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:02:24.558738 (+ 205us) client_negotiation.cc:769] Sending connection context
0929 19:02:24.558860 (+ 122us) client_negotiation.cc:240] Negotiation successful
0929 19:02:24.559020 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":94,"threads_started":1}
I20260929 19:02:24.559947 11835 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33605
I20260929 19:02:24.560040 11835 heartbeater.cc:464] Registering TS with master...
I20260929 19:02:24.560217 11835 heartbeater.cc:511] Master 127.0.98.62:33605 requested a full tablet report, sending...
I20260929 19:02:24.560701 11623 ts_manager.cc:194] Registered new tserver with Master: f14b54af233143cf85bbff7c9bb1dbc8 (127.0.98.1:45205)
I20260929 19:02:24.561296 11623 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:55775
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708447806542-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708447806542-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260929 19:02:24.582832 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUbnLHE/build/release/bin/kudu
/tmp/dist-test-taskUbnLHE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708447806542-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708447806542-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708447806542-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708447806542-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:52183/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33605
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708447806542-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708447806542-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708447806542-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708447806542-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:02:24.689960 11841 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:02:24.690214 11841 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:02:24.690248 11841 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:02:24.691776 11841 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:02:24.691830 11841 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:02:24.691866 11841 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 19:02:24.693859 11841 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708447806542-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708447806542-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:52183/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708447806542-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708447806542-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33605
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11841
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708447806542-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 18:09:13 UTC on ccycloud.quasar-hpkalf.root.comops.site
build id 4680913
I20260929 19:02:24.694075 11841 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:02:24.694367 11841 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:02:24.700569 11849 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:02:24.700608 11848 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:02:24.700989 11851 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:02:24.701293 11841 server_base.cc:1034] running on GCE node
I20260929 19:02:24.701478 11841 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:02:24.701731 11841 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:02:24.702903 11841 hybrid_clock.cc:648] HybridClock initialized: now 1790708544702882 us; error 32 us; skew 500 ppm
Sep 29 19:02:24 dist-test-slave-lqkv krb5kdc[11147](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708544, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:02:24.705893 11841 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 19:02:24.706342 11841 webserver.cc:460] Webserver started at http://127.0.98.2:37659/ using document root <none> and password file <none>
I20260929 19:02:24.706553 11841 fs_manager.cc:359] Metadata directory not provided
I20260929 19:02:24.706609 11841 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:02:24.706720 11841 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 19:02:24 dist-test-slave-lqkv krb5kdc[11147](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708544, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 29 19:02:24 dist-test-slave-lqkv krb5kdc[11147](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790708544, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:02:24.722765 11841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708447806542-392-0/minicluster-data/ts-1/data/instance:
uuid: "32026fc075494c0394b3114baad05896"
format_stamp: "Formatted at 2026-09-29 19:02:24 on dist-test-slave-lqkv"
server_key: "c697b54b654c7a2f144b273ef2a3b2b6"
server_key_iv: "c7a2ea041c8bcf5766b8073c260ff6f1"
server_key_version: "kuduclusterkey@0"
I20260929 19:02:24.723147 11841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708447806542-392-0/minicluster-data/ts-1/wal/instance:
uuid: "32026fc075494c0394b3114baad05896"
format_stamp: "Formatted at 2026-09-29 19:02:24 on dist-test-slave-lqkv"
server_key: "c697b54b654c7a2f144b273ef2a3b2b6"
server_key_iv: "c7a2ea041c8bcf5766b8073c260ff6f1"
server_key_version: "kuduclusterkey@0"
I20260929 19:02:24.728093 11841 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260929 19:02:24.739813 11858 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:02:24.741971 11841 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260929 19:02:24.742072 11841 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708447806542-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708447806542-392-0/minicluster-data/ts-1/wal
uuid: "32026fc075494c0394b3114baad05896"
format_stamp: "Formatted at 2026-09-29 19:02:24 on dist-test-slave-lqkv"
server_key: "c697b54b654c7a2f144b273ef2a3b2b6"
server_key_iv: "c7a2ea041c8bcf5766b8073c260ff6f1"
server_key_version: "kuduclusterkey@0"
I20260929 19:02:24.742165 11841 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708447806542-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708447806542-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708447806542-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:02:24.762228 11841 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:02:24.764259 11841 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:02:24.764376 11841 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:02:24.765035 11841 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:02:24.765345 11841 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:02:24.765386 11841 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:02:24.765419 11841 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:02:24.765439 11841 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:02:24.771569 11841 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42221
I20260929 19:02:24.771939 11841 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708447806542-392-0/minicluster-data/ts-1/data/info.pb
I20260929 19:02:24.772472 11971 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42221 every 8 connection(s)
I20260929 19:02:24.778488 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUbnLHE/build/release/bin/kudu as pid 11841
I20260929 19:02:24.778587 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708447806542-392-0/minicluster-data/ts-1/wal/instance
I20260929 19:02:24.778791 392 external_mini_cluster.cc:1428] Setting key ecbd9f614f6650053e610d14d889989c
Sep 29 19:02:24 dist-test-slave-lqkv krb5kdc[11147](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708544, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 19:02:24.786839 11974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:24.772554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33605 (local address 127.0.98.2:51393)
0929 19:02:24.776495 (+ 3941us) negotiation.cc:107] Waiting for socket to connect
0929 19:02:24.776538 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 19:02:24.777121 (+ 583us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:02:24.777372 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:02:24.777376 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:02:24.777572 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:02:24.777829 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:24.777834 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:24.778782 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:24.778783 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:24.780324 (+ 1541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:24.780327 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:24.781057 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:24.781058 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:24.781184 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:24.781754 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:02:24.781763 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:02:24.783213 (+ 1450us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:02:24.785205 (+ 1992us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:02:24.785211 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:02:24.785213 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:02:24.785502 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:02:24.785756 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:02:24.785757 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:02:24.785758 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:02:24.785904 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:02:24.786208 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:02:24.786214 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:02:24.786369 (+ 155us) client_negotiation.cc:769] Sending connection context
0929 19:02:24.786478 (+ 109us) client_negotiation.cc:240] Negotiation successful
0929 19:02:24.786623 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3885,"thread_start_us":82,"threads_started":1}
I20260929 19:02:24.787269 11838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:24.772552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51393 (local address 127.0.98.62:33605)
0929 19:02:24.772599 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:24.772601 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:24.776632 (+ 4031us) server_negotiation.cc:407] Connection header received
0929 19:02:24.777229 (+ 597us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:24.777231 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:24.777242 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:24.777303 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:02:24.777928 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:24.778642 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:24.780422 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:24.780720 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:24.780776 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:24.783319 (+ 2543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:02:24.783320 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:02:24.783320 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:02:24.783328 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:02:24.785112 (+ 1784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:02:24.785590 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:02:24.785592 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:02:24.785592 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:02:24.785664 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:02:24.785996 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:02:24.785997 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:02:24.785998 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:02:24.786127 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:02:24.786197 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:24.787086 (+ 889us) server_negotiation.cc:299] Negotiation successful
0929 19:02:24.787179 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260929 19:02:24.787575 11972 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33605
I20260929 19:02:24.787662 11972 heartbeater.cc:464] Registering TS with master...
I20260929 19:02:24.787822 11972 heartbeater.cc:511] Master 127.0.98.62:33605 requested a full tablet report, sending...
I20260929 19:02:24.788129 11623 ts_manager.cc:194] Registered new tserver with Master: 32026fc075494c0394b3114baad05896 (127.0.98.2:42221)
I20260929 19:02:24.788547 11623 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:51393
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708447806542-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708447806542-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260929 19:02:24.818207 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUbnLHE/build/release/bin/kudu
/tmp/dist-test-taskUbnLHE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708447806542-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708447806542-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708447806542-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708447806542-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:52183/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33605
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708447806542-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708447806542-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708447806542-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708447806542-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:02:24.931236 11978 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:02:24.931442 11978 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:02:24.931478 11978 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:02:24.932523 11978 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:02:24.932586 11978 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:02:24.932628 11978 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 19:02:24.934134 11978 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708447806542-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708447806542-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:52183/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708447806542-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708447806542-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33605
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11978
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708447806542-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 18:09:13 UTC on ccycloud.quasar-hpkalf.root.comops.site
build id 4680913
I20260929 19:02:24.934391 11978 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:02:24.934617 11978 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:02:24.939455 11986 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:02:24.939436 11983 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:02:24.939656 11984 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:02:24.940392 11978 server_base.cc:1034] running on GCE node
I20260929 19:02:24.940591 11978 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:02:24.940866 11978 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:02:24.942034 11978 hybrid_clock.cc:648] HybridClock initialized: now 1790708544942018 us; error 24 us; skew 500 ppm
Sep 29 19:02:24 dist-test-slave-lqkv krb5kdc[11147](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708544, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:02:24.945186 11978 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 19:02:24.945708 11978 webserver.cc:460] Webserver started at http://127.0.98.3:40057/ using document root <none> and password file <none>
I20260929 19:02:24.945927 11978 fs_manager.cc:359] Metadata directory not provided
I20260929 19:02:24.945976 11978 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:02:24.946168 11978 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 19:02:24 dist-test-slave-lqkv krb5kdc[11147](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708544, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260929 19:02:24.968045 11978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708447806542-392-0/minicluster-data/ts-2/data/instance:
uuid: "b3168e0aa3834d90b24fcb0e2330d2d1"
format_stamp: "Formatted at 2026-09-29 19:02:24 on dist-test-slave-lqkv"
server_key: "f2576c01d861f983367b9db57e36de06"
server_key_iv: "8ad66c4494559bddaf71e517ba293db8"
server_key_version: "kuduclusterkey@0"
I20260929 19:02:24.968415 11978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708447806542-392-0/minicluster-data/ts-2/wal/instance:
uuid: "b3168e0aa3834d90b24fcb0e2330d2d1"
format_stamp: "Formatted at 2026-09-29 19:02:24 on dist-test-slave-lqkv"
server_key: "f2576c01d861f983367b9db57e36de06"
server_key_iv: "8ad66c4494559bddaf71e517ba293db8"
server_key_version: "kuduclusterkey@0"
I20260929 19:02:24.969864 11978 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 19:02:24.984658 11993 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:02:24.984881 11978 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260929 19:02:24.984992 11978 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708447806542-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708447806542-392-0/minicluster-data/ts-2/wal
uuid: "b3168e0aa3834d90b24fcb0e2330d2d1"
format_stamp: "Formatted at 2026-09-29 19:02:24 on dist-test-slave-lqkv"
server_key: "f2576c01d861f983367b9db57e36de06"
server_key_iv: "8ad66c4494559bddaf71e517ba293db8"
server_key_version: "kuduclusterkey@0"
I20260929 19:02:24.985102 11978 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708447806542-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708447806542-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708447806542-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:02:25.015161 11978 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:02:25.017504 11978 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:02:25.017652 11978 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:02:25.017880 11978 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:02:25.018235 11978 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:02:25.018285 11978 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:02:25.018328 11978 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:02:25.018355 11978 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:02:25.024923 11978 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45603
I20260929 19:02:25.025332 11978 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708447806542-392-0/minicluster-data/ts-2/data/info.pb
I20260929 19:02:25.025858 12106 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45603 every 8 connection(s)
I20260929 19:02:25.030630 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUbnLHE/build/release/bin/kudu as pid 11978
I20260929 19:02:25.030735 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708447806542-392-0/minicluster-data/ts-2/wal/instance
I20260929 19:02:25.030912 392 external_mini_cluster.cc:1428] Setting key d87d462bf24bd3a91c51b79f541cf42c
Sep 29 19:02:25 dist-test-slave-lqkv krb5kdc[11147](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708544, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 19:02:25.036597 11838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:25.026072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56693 (local address 127.0.98.62:33605)
0929 19:02:25.026112 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:25.026114 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:25.026328 (+ 214us) server_negotiation.cc:407] Connection header received
0929 19:02:25.027428 (+ 1100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:25.027430 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:25.027442 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:25.027481 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:02:25.028261 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:25.029040 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:25.030454 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:25.030777 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:25.030851 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:25.033270 (+ 2419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:02:25.033271 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:02:25.033271 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:02:25.033279 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:02:25.034984 (+ 1705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:02:25.035515 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:02:25.035516 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:02:25.035517 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:02:25.035575 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:02:25.035854 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:02:25.035855 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:02:25.035856 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:02:25.036000 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:02:25.036044 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:25.036427 (+ 383us) server_negotiation.cc:299] Negotiation successful
0929 19:02:25.036509 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260929 19:02:25.037341 12109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:25.026067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33605 (local address 127.0.98.3:56693)
0929 19:02:25.026211 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0929 19:02:25.026253 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 19:02:25.027308 (+ 1055us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:02:25.027667 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:02:25.027671 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:02:25.027897 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:02:25.028151 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:25.028157 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:25.029151 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:25.029152 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:25.030351 (+ 1199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:25.030353 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:25.030916 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:25.030917 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:25.031049 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:25.031585 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:02:25.031595 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:02:25.033176 (+ 1581us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:02:25.035076 (+ 1900us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:02:25.035082 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:02:25.035084 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:02:25.035424 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:02:25.035644 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:02:25.035644 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:02:25.035645 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:02:25.035767 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:02:25.036090 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:02:25.036096 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:02:25.036254 (+ 158us) client_negotiation.cc:769] Sending connection context
0929 19:02:25.036972 (+ 718us) client_negotiation.cc:240] Negotiation successful
0929 19:02:25.037118 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":80,"threads_started":1}
I20260929 19:02:25.037941 12107 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33605
I20260929 19:02:25.038069 12107 heartbeater.cc:464] Registering TS with master...
I20260929 19:02:25.038275 12107 heartbeater.cc:511] Master 127.0.98.62:33605 requested a full tablet report, sending...
I20260929 19:02:25.038678 11623 ts_manager.cc:194] Registered new tserver with Master: b3168e0aa3834d90b24fcb0e2330d2d1 (127.0.98.3:45603)
I20260929 19:02:25.039160 11623 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:56693
I20260929 19:02:25.042412 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 19:02:25.056205 11838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:25.045117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40700 (local address 127.0.98.62:33605)
0929 19:02:25.045152 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:25.045154 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:25.048974 (+ 3820us) server_negotiation.cc:407] Connection header received
0929 19:02:25.049022 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:25.049023 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:25.049033 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:25.049087 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:02:25.049769 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:25.050460 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:25.051525 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:25.051802 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:25.051841 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:25.053274 (+ 1433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:02:25.053276 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:02:25.053276 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:02:25.053282 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:02:25.054967 (+ 1685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:02:25.055373 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:02:25.055374 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:02:25.055374 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:02:25.055437 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:02:25.055662 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:02:25.055663 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:02:25.055663 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:02:25.055788 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:02:25.055891 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:25.056045 (+ 154us) server_negotiation.cc:299] Negotiation successful
0929 19:02:25.056121 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260929 19:02:25.057772 11623 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40700:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 19:02:25.060132 11623 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 19:02:25.068468 12121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:25.062659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45603 (local address 127.0.0.1:35736)
0929 19:02:25.063789 (+ 1130us) negotiation.cc:107] Waiting for socket to connect
0929 19:02:25.063825 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0929 19:02:25.063884 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:02:25.064340 (+ 456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:02:25.064343 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:02:25.064345 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:02:25.064403 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:25.064405 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:25.065773 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:25.065774 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:25.067090 (+ 1316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:25.067093 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:25.068174 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:25.068175 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:25.068257 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:25.068270 (+ 13us) client_negotiation.cc:769] Sending connection context
0929 19:02:25.068351 (+ 81us) client_negotiation.cc:240] Negotiation successful
0929 19:02:25.068397 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1103,"thread_start_us":72,"threads_started":1}
I20260929 19:02:25.068778 12122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:25.063324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35736 (local address 127.0.98.3:45603)
0929 19:02:25.064005 (+ 681us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:25.064010 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:25.064018 (+ 8us) server_negotiation.cc:407] Connection header received
0929 19:02:25.064051 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:25.064053 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:25.064067 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:25.064134 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:02:25.064490 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:25.065679 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:25.067214 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:25.067980 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:25.068055 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:25.068128 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:25.068664 (+ 536us) server_negotiation.cc:299] Negotiation successful
0929 19:02:25.068704 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":652,"thread_start_us":64,"threads_started":1}
I20260929 19:02:25.069497 12026 tablet_service.cc:1467] Processing CreateTablet for tablet b55f1b7c7576416d9b19f374f99ed9cd (DEFAULT_TABLE table=test-table [id=d476743febb8422da656b143648fc0c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:02:25.069830 12026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b55f1b7c7576416d9b19f374f99ed9cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:02:25.074761 12121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:25.063028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42221 (local address 127.0.0.1:37974)
0929 19:02:25.068543 (+ 5515us) negotiation.cc:107] Waiting for socket to connect
0929 19:02:25.068552 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 19:02:25.068593 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:02:25.071539 (+ 2946us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:02:25.071540 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:02:25.071541 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:02:25.071567 (+ 26us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:25.071568 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:25.072705 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:25.072706 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:25.073662 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:25.073665 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:25.074540 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:25.074541 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:25.074597 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:25.074600 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 19:02:25.074662 (+ 62us) client_negotiation.cc:240] Negotiation successful
0929 19:02:25.074706 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5506,"thread_start_us":53,"threads_started":1}
I20260929 19:02:25.075052 12124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:25.063627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37974 (local address 127.0.98.2:42221)
0929 19:02:25.064303 (+ 676us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:25.064306 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:25.068822 (+ 4516us) server_negotiation.cc:407] Connection header received
0929 19:02:25.069133 (+ 311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:25.069135 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:25.069150 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:25.069226 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:02:25.071646 (+ 2420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:25.072617 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:25.073771 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:25.074439 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:25.074521 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:25.074586 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:25.074937 (+ 351us) server_negotiation.cc:299] Negotiation successful
0929 19:02:25.074976 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":649,"thread_start_us":72,"threads_started":1}
I20260929 19:02:25.075088 12120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:25.062859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45205 (local address 127.0.0.1:54412)
0929 19:02:25.067219 (+ 4360us) negotiation.cc:107] Waiting for socket to connect
0929 19:02:25.067229 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 19:02:25.067271 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:02:25.067533 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:02:25.067533 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:02:25.067535 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:02:25.067583 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:25.067585 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:25.069541 (+ 1956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:25.069542 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:25.070615 (+ 1073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:25.070617 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:25.074621 (+ 4004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:25.074622 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:25.074691 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:25.074695 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 19:02:25.074747 (+ 52us) client_negotiation.cc:240] Negotiation successful
0929 19:02:25.074804 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4345,"thread_start_us":69,"threads_started":1}
I20260929 19:02:25.075724 11896 tablet_service.cc:1467] Processing CreateTablet for tablet b55f1b7c7576416d9b19f374f99ed9cd (DEFAULT_TABLE table=test-table [id=d476743febb8422da656b143648fc0c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:02:25.076076 11896 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b55f1b7c7576416d9b19f374f99ed9cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:02:25.079399 12126 tablet_bootstrap.cc:492] T b55f1b7c7576416d9b19f374f99ed9cd P 32026fc075494c0394b3114baad05896: Bootstrap starting.
I20260929 19:02:25.079854 12123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:25.063492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54412 (local address 127.0.98.1:45205)
0929 19:02:25.064235 (+ 743us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:25.064239 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:25.067329 (+ 3090us) server_negotiation.cc:407] Connection header received
0929 19:02:25.067369 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:25.067371 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:25.067387 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:25.067469 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:02:25.067658 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:25.069444 (+ 1786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:25.070703 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:25.071373 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:25.071441 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:25.071504 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:25.079737 (+ 8233us) server_negotiation.cc:299] Negotiation successful
0929 19:02:25.079784 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":717,"thread_start_us":49,"threads_started":1}
I20260929 19:02:25.080348 11764 tablet_service.cc:1467] Processing CreateTablet for tablet b55f1b7c7576416d9b19f374f99ed9cd (DEFAULT_TABLE table=test-table [id=d476743febb8422da656b143648fc0c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:02:25.080644 11764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b55f1b7c7576416d9b19f374f99ed9cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:02:25.080657 12126 tablet_bootstrap.cc:654] T b55f1b7c7576416d9b19f374f99ed9cd P 32026fc075494c0394b3114baad05896: Neither blocks nor log segments found. Creating new log.
I20260929 19:02:25.080981 12125 tablet_bootstrap.cc:492] T b55f1b7c7576416d9b19f374f99ed9cd P b3168e0aa3834d90b24fcb0e2330d2d1: Bootstrap starting.
I20260929 19:02:25.081009 12126 log.cc:824] T b55f1b7c7576416d9b19f374f99ed9cd P 32026fc075494c0394b3114baad05896: Log is configured to *not* fsync() on all Append() calls
I20260929 19:02:25.081773 12126 tablet_bootstrap.cc:492] T b55f1b7c7576416d9b19f374f99ed9cd P 32026fc075494c0394b3114baad05896: No bootstrap required, opened a new log
I20260929 19:02:25.081843 12126 ts_tablet_manager.cc:1397] T b55f1b7c7576416d9b19f374f99ed9cd P 32026fc075494c0394b3114baad05896: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260929 19:02:25.081909 12125 tablet_bootstrap.cc:654] T b55f1b7c7576416d9b19f374f99ed9cd P b3168e0aa3834d90b24fcb0e2330d2d1: Neither blocks nor log segments found. Creating new log.
I20260929 19:02:25.082182 12125 log.cc:824] T b55f1b7c7576416d9b19f374f99ed9cd P b3168e0aa3834d90b24fcb0e2330d2d1: Log is configured to *not* fsync() on all Append() calls
I20260929 19:02:25.084100 12126 raft_consensus.cc:348] T b55f1b7c7576416d9b19f374f99ed9cd P 32026fc075494c0394b3114baad05896 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3168e0aa3834d90b24fcb0e2330d2d1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45603 } } peers { permanent_uuid: "f14b54af233143cf85bbff7c9bb1dbc8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45205 } } peers { permanent_uuid: "32026fc075494c0394b3114baad05896" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42221 } }
I20260929 19:02:25.084254 12126 raft_consensus.cc:374] T b55f1b7c7576416d9b19f374f99ed9cd P 32026fc075494c0394b3114baad05896 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:02:25.084292 12126 raft_consensus.cc:729] T b55f1b7c7576416d9b19f374f99ed9cd P 32026fc075494c0394b3114baad05896 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32026fc075494c0394b3114baad05896, State: Initialized, Role: FOLLOWER
I20260929 19:02:25.084401 12126 consensus_queue.cc:260] T b55f1b7c7576416d9b19f374f99ed9cd P 32026fc075494c0394b3114baad05896 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3168e0aa3834d90b24fcb0e2330d2d1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45603 } } peers { permanent_uuid: "f14b54af233143cf85bbff7c9bb1dbc8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45205 } } peers { permanent_uuid: "32026fc075494c0394b3114baad05896" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42221 } }
I20260929 19:02:25.084630 12126 ts_tablet_manager.cc:1428] T b55f1b7c7576416d9b19f374f99ed9cd P 32026fc075494c0394b3114baad05896: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260929 19:02:25.084877 11972 heartbeater.cc:503] Master 127.0.98.62:33605 was elected leader, sending a full tablet report...
I20260929 19:02:25.087911 12125 tablet_bootstrap.cc:492] T b55f1b7c7576416d9b19f374f99ed9cd P b3168e0aa3834d90b24fcb0e2330d2d1: No bootstrap required, opened a new log
I20260929 19:02:25.087975 12125 ts_tablet_manager.cc:1397] T b55f1b7c7576416d9b19f374f99ed9cd P b3168e0aa3834d90b24fcb0e2330d2d1: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260929 19:02:25.089771 12125 raft_consensus.cc:348] T b55f1b7c7576416d9b19f374f99ed9cd P b3168e0aa3834d90b24fcb0e2330d2d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3168e0aa3834d90b24fcb0e2330d2d1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45603 } } peers { permanent_uuid: "f14b54af233143cf85bbff7c9bb1dbc8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45205 } } peers { permanent_uuid: "32026fc075494c0394b3114baad05896" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42221 } }
I20260929 19:02:25.089906 12125 raft_consensus.cc:374] T b55f1b7c7576416d9b19f374f99ed9cd P b3168e0aa3834d90b24fcb0e2330d2d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:02:25.089937 12125 raft_consensus.cc:729] T b55f1b7c7576416d9b19f374f99ed9cd P b3168e0aa3834d90b24fcb0e2330d2d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3168e0aa3834d90b24fcb0e2330d2d1, State: Initialized, Role: FOLLOWER
I20260929 19:02:25.090037 12125 consensus_queue.cc:260] T b55f1b7c7576416d9b19f374f99ed9cd P b3168e0aa3834d90b24fcb0e2330d2d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3168e0aa3834d90b24fcb0e2330d2d1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45603 } } peers { permanent_uuid: "f14b54af233143cf85bbff7c9bb1dbc8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45205 } } peers { permanent_uuid: "32026fc075494c0394b3114baad05896" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42221 } }
I20260929 19:02:25.090236 12125 ts_tablet_manager.cc:1428] T b55f1b7c7576416d9b19f374f99ed9cd P b3168e0aa3834d90b24fcb0e2330d2d1: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260929 19:02:25.090338 12107 heartbeater.cc:503] Master 127.0.98.62:33605 was elected leader, sending a full tablet report...
I20260929 19:02:25.092955 12129 tablet_bootstrap.cc:492] T b55f1b7c7576416d9b19f374f99ed9cd P f14b54af233143cf85bbff7c9bb1dbc8: Bootstrap starting.
I20260929 19:02:25.093990 12129 tablet_bootstrap.cc:654] T b55f1b7c7576416d9b19f374f99ed9cd P f14b54af233143cf85bbff7c9bb1dbc8: Neither blocks nor log segments found. Creating new log.
I20260929 19:02:25.094252 12129 log.cc:824] T b55f1b7c7576416d9b19f374f99ed9cd P f14b54af233143cf85bbff7c9bb1dbc8: Log is configured to *not* fsync() on all Append() calls
I20260929 19:02:25.094887 12129 tablet_bootstrap.cc:492] T b55f1b7c7576416d9b19f374f99ed9cd P f14b54af233143cf85bbff7c9bb1dbc8: No bootstrap required, opened a new log
I20260929 19:02:25.094951 12129 ts_tablet_manager.cc:1397] T b55f1b7c7576416d9b19f374f99ed9cd P f14b54af233143cf85bbff7c9bb1dbc8: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 19:02:25.097152 12129 raft_consensus.cc:348] T b55f1b7c7576416d9b19f374f99ed9cd P f14b54af233143cf85bbff7c9bb1dbc8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3168e0aa3834d90b24fcb0e2330d2d1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45603 } } peers { permanent_uuid: "f14b54af233143cf85bbff7c9bb1dbc8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45205 } } peers { permanent_uuid: "32026fc075494c0394b3114baad05896" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42221 } }
I20260929 19:02:25.097289 12129 raft_consensus.cc:374] T b55f1b7c7576416d9b19f374f99ed9cd P f14b54af233143cf85bbff7c9bb1dbc8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:02:25.097334 12129 raft_consensus.cc:729] T b55f1b7c7576416d9b19f374f99ed9cd P f14b54af233143cf85bbff7c9bb1dbc8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f14b54af233143cf85bbff7c9bb1dbc8, State: Initialized, Role: FOLLOWER
I20260929 19:02:25.097427 12129 consensus_queue.cc:260] T b55f1b7c7576416d9b19f374f99ed9cd P f14b54af233143cf85bbff7c9bb1dbc8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3168e0aa3834d90b24fcb0e2330d2d1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45603 } } peers { permanent_uuid: "f14b54af233143cf85bbff7c9bb1dbc8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45205 } } peers { permanent_uuid: "32026fc075494c0394b3114baad05896" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42221 } }
I20260929 19:02:25.097628 12129 ts_tablet_manager.cc:1428] T b55f1b7c7576416d9b19f374f99ed9cd P f14b54af233143cf85bbff7c9bb1dbc8: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 19:02:25.097723 11835 heartbeater.cc:503] Master 127.0.98.62:33605 was elected leader, sending a full tablet report...
I20260929 19:02:25.138278 12131 raft_consensus.cc:482] T b55f1b7c7576416d9b19f374f99ed9cd P b3168e0aa3834d90b24fcb0e2330d2d1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 19:02:25.138392 12131 raft_consensus.cc:504] T b55f1b7c7576416d9b19f374f99ed9cd P b3168e0aa3834d90b24fcb0e2330d2d1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3168e0aa3834d90b24fcb0e2330d2d1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45603 } } peers { permanent_uuid: "f14b54af233143cf85bbff7c9bb1dbc8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45205 } } peers { permanent_uuid: "32026fc075494c0394b3114baad05896" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42221 } }
I20260929 19:02:25.138720 12131 leader_election.cc:288] T b55f1b7c7576416d9b19f374f99ed9cd P b3168e0aa3834d90b24fcb0e2330d2d1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f14b54af233143cf85bbff7c9bb1dbc8 (127.0.98.1:45205), 32026fc075494c0394b3114baad05896 (127.0.98.2:42221)
I20260929 19:02:25.142872 12124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:25.139448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34007 (local address 127.0.98.2:42221)
0929 19:02:25.139485 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:25.139487 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:25.139584 (+ 97us) server_negotiation.cc:407] Connection header received
0929 19:02:25.139673 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:25.139674 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:25.139684 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:25.139729 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:02:25.139934 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:25.140659 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:25.142050 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:25.142548 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:25.142588 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:25.142598 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:25.142779 (+ 181us) server_negotiation.cc:299] Negotiation successful
0929 19:02:25.142809 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260929 19:02:25.142979 12109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:25.138941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45205 (local address 127.0.98.3:60425)
0929 19:02:25.139208 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0929 19:02:25.139218 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 19:02:25.139257 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:02:25.139623 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:02:25.139625 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:02:25.139627 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:02:25.139700 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:25.139702 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:25.140622 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:25.140623 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:25.141956 (+ 1333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:25.141958 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:25.142730 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:25.142731 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:25.142801 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:25.142804 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 19:02:25.142846 (+ 42us) client_negotiation.cc:240] Negotiation successful
0929 19:02:25.142880 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"mutex_wait_us":4}
I20260929 19:02:25.143093 12134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:25.139049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42221 (local address 127.0.98.3:34007)
0929 19:02:25.139535 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0929 19:02:25.139543 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0929 19:02:25.139603 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:02:25.139789 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:02:25.139790 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:02:25.139792 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:02:25.139865 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:25.139866 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:25.140767 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:25.140768 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:25.141956 (+ 1188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:02:25.141958 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:02:25.142648 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:02:25.142649 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:02:25.142712 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:25.142715 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 19:02:25.142977 (+ 262us) client_negotiation.cc:240] Negotiation successful
0929 19:02:25.143014 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":82,"threads_started":1}
I20260929 19:02:25.143247 12123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:25.139393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60425 (local address 127.0.98.1:45205)
0929 19:02:25.139445 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:25.139447 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:25.139457 (+ 10us) server_negotiation.cc:407] Connection header received
0929 19:02:25.139495 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:25.139496 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:25.139508 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:25.139568 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:02:25.139795 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:25.140528 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:25.142049 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:25.142603 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:25.142666 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:25.142674 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:25.143142 (+ 468us) server_negotiation.cc:299] Negotiation successful
0929 19:02:25.143194 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260929 19:02:25.143440 11916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b55f1b7c7576416d9b19f374f99ed9cd" candidate_uuid: "b3168e0aa3834d90b24fcb0e2330d2d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32026fc075494c0394b3114baad05896" is_pre_election: true
I20260929 19:02:25.143468 11789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b55f1b7c7576416d9b19f374f99ed9cd" candidate_uuid: "b3168e0aa3834d90b24fcb0e2330d2d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f14b54af233143cf85bbff7c9bb1dbc8" is_pre_election: true
I20260929 19:02:25.143589 11789 raft_consensus.cc:2454] T b55f1b7c7576416d9b19f374f99ed9cd P f14b54af233143cf85bbff7c9bb1dbc8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3168e0aa3834d90b24fcb0e2330d2d1 in term 0.
I20260929 19:02:25.143594 11916 raft_consensus.cc:2454] T b55f1b7c7576416d9b19f374f99ed9cd P 32026fc075494c0394b3114baad05896 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3168e0aa3834d90b24fcb0e2330d2d1 in term 0.
I20260929 19:02:25.143791 11995 leader_election.cc:302] T b55f1b7c7576416d9b19f374f99ed9cd P b3168e0aa3834d90b24fcb0e2330d2d1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b3168e0aa3834d90b24fcb0e2330d2d1, f14b54af233143cf85bbff7c9bb1dbc8; no voters:
I20260929 19:02:25.143925 12131 raft_consensus.cc:2781] T b55f1b7c7576416d9b19f374f99ed9cd P b3168e0aa3834d90b24fcb0e2330d2d1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 19:02:25.143982 12131 raft_consensus.cc:482] T b55f1b7c7576416d9b19f374f99ed9cd P b3168e0aa3834d90b24fcb0e2330d2d1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 19:02:25.144001 12131 raft_consensus.cc:3037] T b55f1b7c7576416d9b19f374f99ed9cd P b3168e0aa3834d90b24fcb0e2330d2d1 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:02:25.144738 12131 raft_consensus.cc:504] T b55f1b7c7576416d9b19f374f99ed9cd P b3168e0aa3834d90b24fcb0e2330d2d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3168e0aa3834d90b24fcb0e2330d2d1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45603 } } peers { permanent_uuid: "f14b54af233143cf85bbff7c9bb1dbc8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45205 } } peers { permanent_uuid: "32026fc075494c0394b3114baad05896" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42221 } }
I20260929 19:02:25.144892 12131 leader_election.cc:288] T b55f1b7c7576416d9b19f374f99ed9cd P b3168e0aa3834d90b24fcb0e2330d2d1 [CANDIDATE]: Term 1 election: Requested vote from peers f14b54af233143cf85bbff7c9bb1dbc8 (127.0.98.1:45205), 32026fc075494c0394b3114baad05896 (127.0.98.2:42221)
I20260929 19:02:25.145112 11789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b55f1b7c7576416d9b19f374f99ed9cd" candidate_uuid: "b3168e0aa3834d90b24fcb0e2330d2d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f14b54af233143cf85bbff7c9bb1dbc8"
I20260929 19:02:25.145123 11916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b55f1b7c7576416d9b19f374f99ed9cd" candidate_uuid: "b3168e0aa3834d90b24fcb0e2330d2d1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32026fc075494c0394b3114baad05896"
I20260929 19:02:25.145184 11916 raft_consensus.cc:3037] T b55f1b7c7576416d9b19f374f99ed9cd P 32026fc075494c0394b3114baad05896 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:02:25.145184 11789 raft_consensus.cc:3037] T b55f1b7c7576416d9b19f374f99ed9cd P f14b54af233143cf85bbff7c9bb1dbc8 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:02:25.145994 11916 raft_consensus.cc:2454] T b55f1b7c7576416d9b19f374f99ed9cd P 32026fc075494c0394b3114baad05896 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3168e0aa3834d90b24fcb0e2330d2d1 in term 1.
I20260929 19:02:25.146198 11996 leader_election.cc:302] T b55f1b7c7576416d9b19f374f99ed9cd P b3168e0aa3834d90b24fcb0e2330d2d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32026fc075494c0394b3114baad05896, b3168e0aa3834d90b24fcb0e2330d2d1; no voters:
I20260929 19:02:25.146236 11789 raft_consensus.cc:2454] T b55f1b7c7576416d9b19f374f99ed9cd P f14b54af233143cf85bbff7c9bb1dbc8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3168e0aa3834d90b24fcb0e2330d2d1 in term 1.
I20260929 19:02:25.146268 12131 raft_consensus.cc:2781] T b55f1b7c7576416d9b19f374f99ed9cd P b3168e0aa3834d90b24fcb0e2330d2d1 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:02:25.146404 12131 raft_consensus.cc:686] T b55f1b7c7576416d9b19f374f99ed9cd P b3168e0aa3834d90b24fcb0e2330d2d1 [term 1 LEADER]: Becoming Leader. State: Replica: b3168e0aa3834d90b24fcb0e2330d2d1, State: Running, Role: LEADER
I20260929 19:02:25.146494 12131 consensus_queue.cc:237] T b55f1b7c7576416d9b19f374f99ed9cd P b3168e0aa3834d90b24fcb0e2330d2d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3168e0aa3834d90b24fcb0e2330d2d1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45603 } } peers { permanent_uuid: "f14b54af233143cf85bbff7c9bb1dbc8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45205 } } peers { permanent_uuid: "32026fc075494c0394b3114baad05896" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42221 } }
I20260929 19:02:25.147190 11623 catalog_manager.cc:5495] T b55f1b7c7576416d9b19f374f99ed9cd P b3168e0aa3834d90b24fcb0e2330d2d1 reported cstate change: term changed from 0 to 1, leader changed from <none> to b3168e0aa3834d90b24fcb0e2330d2d1 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "b3168e0aa3834d90b24fcb0e2330d2d1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3168e0aa3834d90b24fcb0e2330d2d1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45603 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f14b54af233143cf85bbff7c9bb1dbc8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32026fc075494c0394b3114baad05896" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42221 } health_report { overall_health: UNKNOWN } } }
Sep 29 19:02:25 dist-test-slave-lqkv krb5kdc[11147](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708544, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260929 19:02:25.159198 12122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:25.156264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35740 (local address 127.0.98.3:45603)
0929 19:02:25.156297 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0929 19:02:25.156299 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:02:25.156305 (+ 6us) server_negotiation.cc:407] Connection header received
0929 19:02:25.156330 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:02:25.156330 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:02:25.156345 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:02:25.156400 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 19:02:25.156575 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:25.157163 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:25.158188 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:02:25.158504 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:02:25.158564 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:02:25.158798 (+ 234us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 19:02:25.158935 (+ 137us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 19:02:25.159019 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0929 19:02:25.159118 (+ 99us) server_negotiation.cc:299] Negotiation successful
0929 19:02:25.159138 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260929 19:02:25.160925 11916 raft_consensus.cc:1261] T b55f1b7c7576416d9b19f374f99ed9cd P 32026fc075494c0394b3114baad05896 [term 1 FOLLOWER]: Refusing update from remote peer b3168e0aa3834d90b24fcb0e2330d2d1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 19:02:25.160944 11789 raft_consensus.cc:1261] T b55f1b7c7576416d9b19f374f99ed9cd P f14b54af233143cf85bbff7c9bb1dbc8 [term 1 FOLLOWER]: Refusing update from remote peer b3168e0aa3834d90b24fcb0e2330d2d1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 19:02:25.161255 12135 consensus_queue.cc:1035] T b55f1b7c7576416d9b19f374f99ed9cd P b3168e0aa3834d90b24fcb0e2330d2d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f14b54af233143cf85bbff7c9bb1dbc8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 19:02:25.161355 12131 consensus_queue.cc:1035] T b55f1b7c7576416d9b19f374f99ed9cd P b3168e0aa3834d90b24fcb0e2330d2d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32026fc075494c0394b3114baad05896" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 19:02:25.163856 12138 mvcc.cc:205] Tried to move back new op lower bound from 7334742200976609280 to 7334742200920412160. Current Snapshot: MvccSnapshot[applied={T|T < 7334742200976609280}]
I20260929 19:02:25.164184 12140 mvcc.cc:205] Tried to move back new op lower bound from 7334742200976609280 to 7334742200920412160. Current Snapshot: MvccSnapshot[applied={T|T < 7334742200976609280 or (T in {7334742200976609280})}]
I20260929 19:02:25.166337 11623 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40700:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 19:02:25.166416 11623 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40700:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 19:02:25.167055 11623 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b1cbc4fa8c374a72ab0043f6d3b417e1: Sending DeleteTablet for 3 replicas of tablet b55f1b7c7576416d9b19f374f99ed9cd
I20260929 19:02:25.167491 11764 tablet_service.cc:1514] Processing DeleteTablet for tablet b55f1b7c7576416d9b19f374f99ed9cd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:02:25 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54412
I20260929 19:02:25.167516 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUbnLHE/build/release/bin/kudu with pid 11704
I20260929 19:02:25.167951 11896 tablet_service.cc:1514] Processing DeleteTablet for tablet b55f1b7c7576416d9b19f374f99ed9cd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:02:25 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37974
I20260929 19:02:25.168871 12026 tablet_service.cc:1514] Processing DeleteTablet for tablet b55f1b7c7576416d9b19f374f99ed9cd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:02:25 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35736
I20260929 19:02:25.169124 12148 tablet_replica.cc:330] T b55f1b7c7576416d9b19f374f99ed9cd P b3168e0aa3834d90b24fcb0e2330d2d1: stopping tablet replica
I20260929 19:02:25.169203 12148 raft_consensus.cc:2229] T b55f1b7c7576416d9b19f374f99ed9cd P b3168e0aa3834d90b24fcb0e2330d2d1 [term 1 LEADER]: Raft consensus shutting down.
I20260929 19:02:25.169305 12148 raft_consensus.cc:2258] T b55f1b7c7576416d9b19f374f99ed9cd P b3168e0aa3834d90b24fcb0e2330d2d1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 19:02:25.172554 12148 ts_tablet_manager.cc:1905] T b55f1b7c7576416d9b19f374f99ed9cd P b3168e0aa3834d90b24fcb0e2330d2d1: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260929 19:02:25.173991 11608 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:45205 (error 108)
I20260929 19:02:25.174203 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUbnLHE/build/release/bin/kudu with pid 11841
I20260929 19:02:25.175046 12148 ts_tablet_manager.cc:1918] T b55f1b7c7576416d9b19f374f99ed9cd P b3168e0aa3834d90b24fcb0e2330d2d1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 19:02:25.175118 12148 log.cc:1196] T b55f1b7c7576416d9b19f374f99ed9cd P b3168e0aa3834d90b24fcb0e2330d2d1: Deleting WAL directory at /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790708447806542-392-0/minicluster-data/ts-2/wal/wals/b55f1b7c7576416d9b19f374f99ed9cd
I20260929 19:02:25.175419 12148 ts_tablet_manager.cc:1939] T b55f1b7c7576416d9b19f374f99ed9cd P b3168e0aa3834d90b24fcb0e2330d2d1: Deleting consensus metadata
I20260929 19:02:25.175776 11608 catalog_manager.cc:4841] TS b3168e0aa3834d90b24fcb0e2330d2d1 (127.0.98.3:45603): tablet b55f1b7c7576416d9b19f374f99ed9cd (table test-table [id=d476743febb8422da656b143648fc0c2]) successfully deleted
I20260929 19:02:25.181951 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUbnLHE/build/release/bin/kudu with pid 11978
I20260929 19:02:25.186154 12120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:25.183348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45205 (local address 127.0.0.1:54414)
0929 19:02:25.186048 (+ 2700us) negotiation.cc:107] Waiting for socket to connect
0929 19:02:25.186122 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45205: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2680}
I20260929 19:02:25.186268 12120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:02:25.183428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42221 (local address 127.0.0.1:37990)
0929 19:02:25.186238 (+ 2810us) negotiation.cc:107] Waiting for socket to connect
0929 19:02:25.186257 (+ 19us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:42221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2801}
W20260929 19:02:25.186456 11608 catalog_manager.cc:4584] TS f14b54af233143cf85bbff7c9bb1dbc8 (127.0.98.1:45205): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b55f1b7c7576416d9b19f374f99ed9cd: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45205: connect: Connection refused (error 111)
I20260929 19:02:25.188143 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUbnLHE/build/release/bin/kudu with pid 11591
I20260929 19:02:25.199561 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260929 19:02:25.544693 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260929 19:02:25.544786 392 mini_ranger.cc:66] Stopping Ranger...
I20260929 19:02:25.891367 392 mini_ranger.cc:68] Stopped Ranger
2026-09-29 19:02:25.891 UTC [11177] LOG: received smart shutdown request
2026-09-29 19:02:25.894 UTC [11177] LOG: background worker "logical replication launcher" (PID 11186) exited with exit code 1
2026-09-29 19:02:25.894 UTC [11181] LOG: shutting down
2026-09-29 19:02:26.051 UTC [11177] LOG: database system is shut down
2026-09-29T19:02:26Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (42188 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 19:02:26 dist-test-slave-lqkv krb5kdc[12158](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 19:02:26 dist-test-slave-lqkv krb5kdc[12158](info): set up 2 sockets
Sep 29 19:02:26 dist-test-slave-lqkv krb5kdc[12158](info): commencing operation
krb5kdc: starting...
W20260929 19:02:27.148574 392 mini_kdc.cc:121] Time spent starting KDC: real 0.979s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/krb5kdc/test-user.keytab.
Sep 29 19:02:27 dist-test-slave-lqkv krb5kdc[12158](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708547, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T19:02:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T19:02:27Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260929 19:02:27.295424 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskUbnLHE/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/minicluster-data/postgres -l logfile start
2026-09-29 19:02:29.637 UTC [12188] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-29 19:02:29.637 UTC [12188] LOG: listening on IPv4 address "127.0.98.20", port 60575
2026-09-29 19:02:29.640 UTC [12188] LOG: listening on Unix socket "/tmp/.s.PGSQL.60575"
2026-09-29 19:02:29.673 UTC [12191] LOG: database system was shut down at 2026-09-29 19:02:28 UTC
2026-09-29 19:02:29.676 UTC [12188] LOG: database system is ready to accept connections
I20260929 19:02:30.584415 392 mini_postgres.cc:96] Postgres bound to 60575
2026-09-29 19:02:30.589 UTC [12199] FATAL: database "slave" does not exist
127.0.98.20:60575 - accepting connections
I20260929 19:02:30.590394 392 mini_ranger.cc:161] Starting Ranger...
I20260929 19:02:30.604197 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260929 19:02:31.058256 392 mini_ranger.cc:87] Created ranger Postgres database
I20260929 19:02:31.058339 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/minicluster-data/ranger-admin
2026-09-29 19:02:31,306 [I] DB FLAVOR :POSTGRES
2026-09-29 19:02:31,307 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 19:02:31,307 [I] Checking connection..
2026-09-29 19:02:31,307 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUbnLHE/build/release/bin/postgresql.jar:/tmp/dist-test-taskUbnLHE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:60575/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-29 19:02:31,544 [I] Checking connection passed.
2026-09-29 19:02:31,544 [I] --------- Verifying version history table ---------
2026-09-29 19:02:31,544 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUbnLHE/build/release/bin/postgresql.jar:/tmp/dist-test-taskUbnLHE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:60575/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 19:02:31,789 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 19:02:31,789 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUbnLHE/build/release/bin/postgresql.jar:/tmp/dist-test-taskUbnLHE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:60575/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 19:02:32,017 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 19:02:32,018 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-29 19:02:32,018 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUbnLHE/build/release/bin/postgresql.jar:/tmp/dist-test-taskUbnLHE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:60575/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskUbnLHE/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-29 19:02:32.255 UTC [12263] WARNING: there is no transaction in progress
2026-09-29 19:02:32,261 [I] create_dbversion_catalog.sql file imported successfully
2026-09-29 19:02:32,261 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUbnLHE/build/release/bin/postgresql.jar:/tmp/dist-test-taskUbnLHE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:60575/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 19:02:32,491 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-29 19:02:32,491 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 19:02:32,491 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUbnLHE/build/release/bin/postgresql.jar:/tmp/dist-test-taskUbnLHE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:60575/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-29 19:02:32,731 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUbnLHE/build/release/bin/postgresql.jar:/tmp/dist-test-taskUbnLHE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:60575/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-29 19:02:32,972 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUbnLHE/build/release/bin/postgresql.jar:/tmp/dist-test-taskUbnLHE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:60575/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-lqkv.c.gcp-upstream.internal','N') ;"
2026-09-29 19:02:33,195 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUbnLHE/build/release/bin/postgresql.jar:/tmp/dist-test-taskUbnLHE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:60575/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 19:02:33,424 [I] Table x_portal_user does not exist in database ranger
2026-09-29 19:02:33,424 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUbnLHE/build/release/bin/postgresql.jar:/tmp/dist-test-taskUbnLHE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:60575/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 19:02:33,660 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-29 19:02:33,660 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUbnLHE/build/release/bin/postgresql.jar:/tmp/dist-test-taskUbnLHE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:60575/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 19:02:33,877 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUbnLHE/build/release/bin/postgresql.jar:/tmp/dist-test-taskUbnLHE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:60575/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 19:02:34,116 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-29 19:02:34,116 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUbnLHE/build/release/bin/postgresql.jar:/tmp/dist-test-taskUbnLHE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:60575/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskUbnLHE/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-29 19:02:35.008 UTC [12371] WARNING: there is no transaction in progress
2026-09-29 19:02:35.022 UTC [12371] WARNING: there is no transaction in progress
2026-09-29 19:02:35.038 UTC [12371] WARNING: there is no transaction in progress
2026-09-29 19:02:35.052 UTC [12371] WARNING: there is no transaction in progress
2026-09-29 19:02:35.067 UTC [12371] WARNING: there is no transaction in progress
2026-09-29 19:02:35.084 UTC [12371] WARNING: there is no transaction in progress
2026-09-29 19:02:35.134 UTC [12371] WARNING: there is no transaction in progress
2026-09-29 19:02:35.140 UTC [12371] WARNING: there is no transaction in progress
2026-09-29 19:02:35.154 UTC [12371] WARNING: there is no transaction in progress
2026-09-29 19:02:35.163 UTC [12371] WARNING: there is no transaction in progress
2026-09-29 19:02:35.173 UTC [12371] WARNING: there is no transaction in progress
2026-09-29 19:02:35.185 UTC [12371] WARNING: there is no transaction in progress
2026-09-29 19:02:35.194 UTC [12371] WARNING: there is no transaction in progress
2026-09-29 19:02:35.199 UTC [12371] WARNING: there is no transaction in progress
2026-09-29 19:02:35.205 UTC [12371] WARNING: there is no transaction in progress
2026-09-29 19:02:35.212 UTC [12371] WARNING: there is no transaction in progress
2026-09-29 19:02:35,810 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-29 19:02:35,811 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUbnLHE/build/release/bin/postgresql.jar:/tmp/dist-test-taskUbnLHE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:60575/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-29 19:02:36,022 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-29 19:02:36,022 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUbnLHE/build/release/bin/postgresql.jar:/tmp/dist-test-taskUbnLHE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:60575/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 19:02:36,238 [I] Table x_portal_user already exists in database 'ranger'
2026-09-29 19:02:36,238 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUbnLHE/build/release/bin/postgresql.jar:/tmp/dist-test-taskUbnLHE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:60575/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 19:02:36,477 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-29 19:02:36,477 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUbnLHE/build/release/bin/postgresql.jar:/tmp/dist-test-taskUbnLHE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:60575/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 19:02:36,718 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUbnLHE/build/release/bin/postgresql.jar:/tmp/dist-test-taskUbnLHE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:60575/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 19:02:36,934 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUbnLHE/build/release/bin/postgresql.jar:/tmp/dist-test-taskUbnLHE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:60575/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-lqkv.c.gcp-upstream.internal';"
2026-09-29 19:02:37,155 [I] CORE_DB_SCHEMA import status has been updated
2026-09-29 19:02:37,155 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUbnLHE/build/release/bin/postgresql.jar:/tmp/dist-test-taskUbnLHE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:60575/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-29 19:02:37,395 [I] DB_PATCHES have already been applied
I20260929 19:02:37.401140 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskUbnLHE/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskUbnLHE/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskUbnLHE/build/release/bin/postgresql.jar:/tmp/dist-test-taskUbnLHE/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260929 19:02:37.401197 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260929 19:02:37.404206 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:48049
Sep 29, 2026 7:02:37 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 7:02:37 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 7:02:37 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/minicluster-data/ranger-admin/ews/webapp
Sep 29, 2026 7:02:37 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 29, 2026 7:02:37 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 29, 2026 7:02:38 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/minicluster-data/ranger-admin/ews/webapp].
Sep 29, 2026 7:02:38 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260929 19:02:38.498147 392 mini_ranger.cc:160] Time spent starting Ranger: real 7.908s user 0.000s sys 0.008s
Sep 29, 2026 7:02:55 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 29, 2026 7:02:55 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 29, 2026 7:02:55 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 29, 2026 7:02:55 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 29, 2026 7:02:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 29, 2026 7:02:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 29, 2026 7:02:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 29, 2026 7:02:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 29, 2026 7:02:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 29, 2026 7:02:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 29, 2026 7:02:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 29, 2026 7:02:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 29, 2026 7:02:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 29, 2026 7:02:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 29, 2026 7:02:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 29, 2026 7:02:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 29, 2026 7:02:55 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 29, 2026 7:02:56 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260929 19:02:57.143186 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 29 19:02:57 dist-test-slave-lqkv krb5kdc[12158](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708577, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260929 19:02:57.231859 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260929 19:02:57.240778 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260929 19:02:57.993857 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260929 19:02:57.993928 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/minicluster-data/ranger-kms
I20260929 19:02:57.993944 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:60575
2026-09-29 19:02:58,144 [I] DB FLAVOR :POSTGRES
2026-09-29 19:02:58,145 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 19:02:58,145 [I] Checking connection
2026-09-29 19:02:58,145 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUbnLHE/build/release/bin/postgresql.jar:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:60575/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-29 19:02:58,380 [I] connection success
2026-09-29 19:02:58,380 [I] --------- Verifying Ranger DB tables ---------
2026-09-29 19:02:58,380 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-29 19:02:58,380 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUbnLHE/build/release/bin/postgresql.jar:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:60575/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-29 19:02:58,639 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-29 19:02:58,639 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 19:02:58,640 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-29 19:02:58,640 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUbnLHE/build/release/bin/postgresql.jar:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:60575/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-29 19:02:58,926 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260929 19:02:59.319371 392 mini_ranger_kms.cc:310] Created kms service
I20260929 19:02:59.515300 392 mini_ranger_kms.cc:326] Created kudu user
I20260929 19:02:59.556546 392 mini_ranger_kms.cc:343] Created rangerkms user
Sep 29 19:02:59 dist-test-slave-lqkv krb5kdc[12158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708577, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260929 19:02:59.933252 392 mini_ranger_kms.cc:384] Added ranger policy
I20260929 19:02:59.933362 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskUbnLHE/build/release/bin/postgresql.jar:/tmp/dist-test-taskUbnLHE/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskUbnLHE/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260929 19:02:59.933382 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260929 19:02:59.937021 392 mini_ranger_kms.cc:276] Ranger KMS PID: 12551
I20260929 19:02:59.937104 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:56777
Sep 29, 2026 7:03:01 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 7:03:01 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 7:03:01 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 29, 2026 7:03:01 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 29, 2026 7:03:01 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/minicluster-data/ranger-kms/ews/webapp].
Sep 29, 2026 7:03:01 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260929 19:03:02.110455 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.879s user 0.000s sys 0.135s
I20260929 19:03:02.110546 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260929 19:03:02.110605 392 mini_ranger_kms.cc:401] 127.0.98.20:56777/kms/v1/keys
Sep 29 19:03:02 dist-test-slave-lqkv krb5kdc[12158](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790708582, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 29 19:03:02 dist-test-slave-lqkv krb5kdc[12158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708582, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 29 19:03:03 dist-test-slave-lqkv krb5kdc[12158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708582, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 29 19:03:05 dist-test-slave-lqkv krb5kdc[12158](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708585, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 19:03:05.095404 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUbnLHE/build/release/bin/kudu
/tmp/dist-test-taskUbnLHE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37509
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37509
--ranger_config_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:56777/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:03:05.168377 12602 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:03:05.168572 12602 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:03:05.168604 12602 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:03:05.169672 12602 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 19:03:05.169728 12602 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:03:05.169750 12602 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:03:05.169768 12602 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 19:03:05.169818 12602 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 19:03:05.171255 12602 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:56777/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37509
--ranger_config_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37509
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12602
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 18:09:13 UTC on ccycloud.quasar-hpkalf.root.comops.site
build id 4680913
I20260929 19:03:05.171487 12602 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:03:05.171679 12602 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:03:05.176195 12608 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:03:05.176186 12607 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:03:05.176184 12610 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:03:05.176537 12602 server_base.cc:1034] running on GCE node
I20260929 19:03:05.176787 12602 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:03:05.177116 12602 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:03:05.178274 12602 hybrid_clock.cc:648] HybridClock initialized: now 1790708585178262 us; error 30 us; skew 500 ppm
Sep 29 19:03:05 dist-test-slave-lqkv krb5kdc[12158](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708585, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:03:05.180490 12602 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 19:03:05.180913 12602 webserver.cc:460] Webserver started at http://127.0.98.62:37231/ using document root <none> and password file <none>
I20260929 19:03:05.181105 12602 fs_manager.cc:359] Metadata directory not provided
I20260929 19:03:05.181140 12602 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:03:05.181249 12602 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 19:03:05 dist-test-slave-lqkv krb5kdc[12158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708585, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260929 19:03:05.232292 12602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/minicluster-data/master-0/data/instance:
uuid: "5ed01c77975c4dc98943ebf46fb3f54f"
format_stamp: "Formatted at 2026-09-29 19:03:05 on dist-test-slave-lqkv"
server_key: "5da362bfed92e1edd1f98cc9c4e75c2c"
server_key_iv: "043af3e2bf8b1cf6a9fe9987019a1302"
server_key_version: "kuduclusterkey@0"
I20260929 19:03:05.232686 12602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/minicluster-data/master-0/wal/instance:
uuid: "5ed01c77975c4dc98943ebf46fb3f54f"
format_stamp: "Formatted at 2026-09-29 19:03:05 on dist-test-slave-lqkv"
server_key: "5da362bfed92e1edd1f98cc9c4e75c2c"
server_key_iv: "043af3e2bf8b1cf6a9fe9987019a1302"
server_key_version: "kuduclusterkey@0"
I20260929 19:03:05.234719 12602 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.001s
I20260929 19:03:05.249666 12617 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:05.249897 12602 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260929 19:03:05.249979 12602 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/minicluster-data/master-0/wal
uuid: "5ed01c77975c4dc98943ebf46fb3f54f"
format_stamp: "Formatted at 2026-09-29 19:03:05 on dist-test-slave-lqkv"
server_key: "5da362bfed92e1edd1f98cc9c4e75c2c"
server_key_iv: "043af3e2bf8b1cf6a9fe9987019a1302"
server_key_version: "kuduclusterkey@0"
I20260929 19:03:05.250056 12602 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:03:05.266594 12602 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:03:05.268500 12602 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:03:05.268636 12602 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:03:05.273011 12602 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37509
I20260929 19:03:05.273128 12669 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37509 every 8 connection(s)
I20260929 19:03:05.273386 12602 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/minicluster-data/master-0/data/info.pb
I20260929 19:03:05.274048 12670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:03:05.276089 12670 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ed01c77975c4dc98943ebf46fb3f54f: Bootstrap starting.
I20260929 19:03:05.276631 12670 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5ed01c77975c4dc98943ebf46fb3f54f: Neither blocks nor log segments found. Creating new log.
I20260929 19:03:05.276950 12670 log.cc:824] T 00000000000000000000000000000000 P 5ed01c77975c4dc98943ebf46fb3f54f: Log is configured to *not* fsync() on all Append() calls
I20260929 19:03:05.277601 12670 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ed01c77975c4dc98943ebf46fb3f54f: No bootstrap required, opened a new log
I20260929 19:03:05.279452 12670 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5ed01c77975c4dc98943ebf46fb3f54f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ed01c77975c4dc98943ebf46fb3f54f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37509 } }
I20260929 19:03:05.279579 12670 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5ed01c77975c4dc98943ebf46fb3f54f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:03:05.279600 12670 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5ed01c77975c4dc98943ebf46fb3f54f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ed01c77975c4dc98943ebf46fb3f54f, State: Initialized, Role: FOLLOWER
I20260929 19:03:05.279739 12670 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5ed01c77975c4dc98943ebf46fb3f54f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ed01c77975c4dc98943ebf46fb3f54f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37509 } }
I20260929 19:03:05.279808 12670 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5ed01c77975c4dc98943ebf46fb3f54f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 19:03:05.279878 12670 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5ed01c77975c4dc98943ebf46fb3f54f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 19:03:05.279927 12670 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5ed01c77975c4dc98943ebf46fb3f54f [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:03:05.280287 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUbnLHE/build/release/bin/kudu as pid 12602
I20260929 19:03:05.280398 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/minicluster-data/master-0/wal/instance
I20260929 19:03:05.280503 12670 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5ed01c77975c4dc98943ebf46fb3f54f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ed01c77975c4dc98943ebf46fb3f54f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37509 } }
I20260929 19:03:05.280580 392 external_mini_cluster.cc:1428] Setting key 77894895c7b8cbc7fbd3a6e3eecd7606
I20260929 19:03:05.280586 12670 leader_election.cc:302] T 00000000000000000000000000000000 P 5ed01c77975c4dc98943ebf46fb3f54f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5ed01c77975c4dc98943ebf46fb3f54f; no voters:
I20260929 19:03:05.280735 12670 leader_election.cc:288] T 00000000000000000000000000000000 P 5ed01c77975c4dc98943ebf46fb3f54f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 19:03:05.280761 12673 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5ed01c77975c4dc98943ebf46fb3f54f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:03:05.280908 12673 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5ed01c77975c4dc98943ebf46fb3f54f [term 1 LEADER]: Becoming Leader. State: Replica: 5ed01c77975c4dc98943ebf46fb3f54f, State: Running, Role: LEADER
I20260929 19:03:05.280984 12670 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5ed01c77975c4dc98943ebf46fb3f54f [sys.catalog]: configured and running, proceeding with master startup.
I20260929 19:03:05.280982 12673 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5ed01c77975c4dc98943ebf46fb3f54f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ed01c77975c4dc98943ebf46fb3f54f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37509 } }
I20260929 19:03:05.282927 12670 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260929 19:03:05.282928 12674 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ed01c77975c4dc98943ebf46fb3f54f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5ed01c77975c4dc98943ebf46fb3f54f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ed01c77975c4dc98943ebf46fb3f54f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37509 } } }
I20260929 19:03:05.283270 12674 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ed01c77975c4dc98943ebf46fb3f54f [sys.catalog]: This master's current role is: LEADER
I20260929 19:03:05.282611 12673 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ed01c77975c4dc98943ebf46fb3f54f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5ed01c77975c4dc98943ebf46fb3f54f. Latest consensus state: current_term: 1 leader_uuid: "5ed01c77975c4dc98943ebf46fb3f54f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ed01c77975c4dc98943ebf46fb3f54f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37509 } } }
I20260929 19:03:05.283540 12673 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ed01c77975c4dc98943ebf46fb3f54f [sys.catalog]: This master's current role is: LEADER
I20260929 19:03:05.287250 12690 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 19:03:05.287760 12690 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 19:03:05.289552 12690 catalog_manager.cc:1347] Generated new cluster ID: 8552eba881a545f2a4f236f5b2bffa0f
I20260929 19:03:05.289594 12690 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 29 19:03:05 dist-test-slave-lqkv krb5kdc[12158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708585, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 19:03:05.295071 12676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:05.280985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40278 (local address 127.0.98.62:37509)
0929 19:03:05.281517 (+ 532us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:05.281526 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:05.281557 (+ 31us) server_negotiation.cc:407] Connection header received
0929 19:03:05.281670 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:05.281676 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:05.281755 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:05.281893 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:05.283059 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:05.286811 (+ 3752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:05.287937 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:05.288556 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:05.288647 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:05.291287 (+ 2640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:03:05.291291 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:03:05.291292 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:03:05.291302 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:03:05.293126 (+ 1824us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:05.293491 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:05.293495 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:05.293495 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:05.293604 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:05.293848 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:05.293848 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:05.293849 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:05.294130 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:03:05.294253 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:05.294718 (+ 465us) server_negotiation.cc:299] Negotiation successful
0929 19:03:05.294855 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":484,"thread_start_us":70,"threads_started":1}
I20260929 19:03:05.302421 12690 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 19:03:05.303061 12690 catalog_manager.cc:1504] Loading token signing keys...
I20260929 19:03:05.308706 12690 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5ed01c77975c4dc98943ebf46fb3f54f: Generated new TSK 0
I20260929 19:03:05.308935 12690 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260929 19:03:05.909657 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUbnLHE/build/release/bin/kudu
/tmp/dist-test-taskUbnLHE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:56777/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37509
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:03:06.013440 12715 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:03:06.013638 12715 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:03:06.013670 12715 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:03:06.014851 12715 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:03:06.014914 12715 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:03:06.014956 12715 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 19:03:06.016395 12715 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:56777/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37509
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12715
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 18:09:13 UTC on ccycloud.quasar-hpkalf.root.comops.site
build id 4680913
I20260929 19:03:06.016583 12715 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:03:06.016826 12715 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:03:06.022560 12721 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:03:06.022714 12720 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:03:06.022564 12723 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:03:06.022899 12715 server_base.cc:1034] running on GCE node
I20260929 19:03:06.023088 12715 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:03:06.023334 12715 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:03:06.024503 12715 hybrid_clock.cc:648] HybridClock initialized: now 1790708586024474 us; error 40 us; skew 500 ppm
Sep 29 19:03:06 dist-test-slave-lqkv krb5kdc[12158](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708586, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:03:06.027520 12715 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 19:03:06.027952 12715 webserver.cc:460] Webserver started at http://127.0.98.1:33915/ using document root <none> and password file <none>
I20260929 19:03:06.028163 12715 fs_manager.cc:359] Metadata directory not provided
I20260929 19:03:06.028218 12715 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:03:06.028328 12715 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 19:03:06 dist-test-slave-lqkv krb5kdc[12158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708586, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260929 19:03:06.041061 12715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/minicluster-data/ts-0/data/instance:
uuid: "e5dca27a04a448bb82a8bcaeeae3f252"
format_stamp: "Formatted at 2026-09-29 19:03:06 on dist-test-slave-lqkv"
server_key: "ae0c119149d45e66ff47b2ae47374f0e"
server_key_iv: "75b4b306e1d823baa45e7d2e23ba8dcf"
server_key_version: "kuduclusterkey@0"
I20260929 19:03:06.041386 12715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/minicluster-data/ts-0/wal/instance:
uuid: "e5dca27a04a448bb82a8bcaeeae3f252"
format_stamp: "Formatted at 2026-09-29 19:03:06 on dist-test-slave-lqkv"
server_key: "ae0c119149d45e66ff47b2ae47374f0e"
server_key_iv: "75b4b306e1d823baa45e7d2e23ba8dcf"
server_key_version: "kuduclusterkey@0"
I20260929 19:03:06.042531 12715 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 19:03:06.053226 12730 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:06.053378 12715 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:06.053474 12715 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/minicluster-data/ts-0/wal
uuid: "e5dca27a04a448bb82a8bcaeeae3f252"
format_stamp: "Formatted at 2026-09-29 19:03:06 on dist-test-slave-lqkv"
server_key: "ae0c119149d45e66ff47b2ae47374f0e"
server_key_iv: "75b4b306e1d823baa45e7d2e23ba8dcf"
server_key_version: "kuduclusterkey@0"
I20260929 19:03:06.053557 12715 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:03:06.064745 12715 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:03:06.067101 12715 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:03:06.067222 12715 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:03:06.067428 12715 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:03:06.067721 12715 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:03:06.067756 12715 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:06.067782 12715 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:03:06.067799 12715 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:06.073277 12715 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44183
I20260929 19:03:06.073633 12715 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/minicluster-data/ts-0/data/info.pb
I20260929 19:03:06.077216 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUbnLHE/build/release/bin/kudu as pid 12715
I20260929 19:03:06.077314 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/minicluster-data/ts-0/wal/instance
I20260929 19:03:06.077534 392 external_mini_cluster.cc:1428] Setting key 84263bbb63fe744cd56d98846d1d6524
I20260929 19:03:06.089267 12845 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44183 every 8 connection(s)
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Sep 29 19:03:06 dist-test-slave-lqkv krb5kdc[12158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708586, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 19:03:06.094471 12848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:06.077979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37509 (local address 127.0.98.1:44985)
0929 19:03:06.084987 (+ 7008us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:06.085041 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:06.085575 (+ 534us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:06.085892 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:06.085897 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:06.086117 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:03:06.086389 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:06.086395 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:06.087452 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:06.087453 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:06.088689 (+ 1236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:06.088692 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:06.089261 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:06.089262 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:06.089401 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:06.089905 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:03:06.089915 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:03:06.091286 (+ 1371us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:03:06.093069 (+ 1783us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:03:06.093074 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:03:06.093076 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:03:06.093303 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:03:06.093496 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:03:06.093497 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:03:06.093497 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:03:06.093600 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:03:06.093837 (+ 237us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:03:06.093841 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:03:06.094007 (+ 166us) client_negotiation.cc:769] Sending connection context
0929 19:03:06.094105 (+ 98us) client_negotiation.cc:240] Negotiation successful
0929 19:03:06.094239 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6968,"thread_start_us":67,"threads_started":1}
I20260929 19:03:06.094792 12849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:06.078175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44985 (local address 127.0.98.62:37509)
0929 19:03:06.084920 (+ 6745us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:06.084923 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:06.085691 (+ 768us) server_negotiation.cc:407] Connection header received
0929 19:03:06.085715 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:06.085716 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:06.085728 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:06.085786 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:06.086491 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:06.087336 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:06.088782 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:06.089110 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:06.089168 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:06.091389 (+ 2221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:03:06.091390 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:03:06.091391 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:03:06.091397 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:03:06.092995 (+ 1598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:06.093381 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:06.093382 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:06.093382 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:06.093434 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:06.093658 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:06.093659 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:06.093659 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:06.093769 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:03:06.093817 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:06.094602 (+ 785us) server_negotiation.cc:299] Negotiation successful
0929 19:03:06.094687 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6715,"thread_start_us":61,"threads_started":1}
I20260929 19:03:06.095217 12846 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37509
I20260929 19:03:06.095294 12846 heartbeater.cc:464] Registering TS with master...
I20260929 19:03:06.095448 12846 heartbeater.cc:511] Master 127.0.98.62:37509 requested a full tablet report, sending...
I20260929 19:03:06.095927 12634 ts_manager.cc:194] Registered new tserver with Master: e5dca27a04a448bb82a8bcaeeae3f252 (127.0.98.1:44183)
I20260929 19:03:06.096668 12634 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:44985
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260929 19:03:06.115796 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUbnLHE/build/release/bin/kudu
/tmp/dist-test-taskUbnLHE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:56777/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37509
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:03:06.221015 12852 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:03:06.221320 12852 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:03:06.221411 12852 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:03:06.222956 12852 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:03:06.223042 12852 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:03:06.223107 12852 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 19:03:06.225212 12852 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:56777/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37509
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12852
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 18:09:13 UTC on ccycloud.quasar-hpkalf.root.comops.site
build id 4680913
I20260929 19:03:06.225677 12852 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:03:06.225937 12852 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:03:06.231899 12860 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:03:06.232191 12862 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:03:06.232441 12859 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:03:06.233469 12852 server_base.cc:1034] running on GCE node
I20260929 19:03:06.233650 12852 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:03:06.233872 12852 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:03:06.235041 12852 hybrid_clock.cc:648] HybridClock initialized: now 1790708586235021 us; error 31 us; skew 500 ppm
Sep 29 19:03:06 dist-test-slave-lqkv krb5kdc[12158](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708586, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:03:06.238106 12852 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 19:03:06.238565 12852 webserver.cc:460] Webserver started at http://127.0.98.2:37787/ using document root <none> and password file <none>
I20260929 19:03:06.238783 12852 fs_manager.cc:359] Metadata directory not provided
I20260929 19:03:06.238831 12852 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:03:06.238960 12852 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 19:03:06 dist-test-slave-lqkv krb5kdc[12158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708586, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260929 19:03:06.252307 12852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/minicluster-data/ts-1/data/instance:
uuid: "6f7bc5348fe64b448f7da8c97840df38"
format_stamp: "Formatted at 2026-09-29 19:03:06 on dist-test-slave-lqkv"
server_key: "fd3dae6793ef6d9a07e5b40254172bdb"
server_key_iv: "e27f142e70d57f0ffbaa608aa431a14d"
server_key_version: "kuduclusterkey@0"
I20260929 19:03:06.252734 12852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/minicluster-data/ts-1/wal/instance:
uuid: "6f7bc5348fe64b448f7da8c97840df38"
format_stamp: "Formatted at 2026-09-29 19:03:06 on dist-test-slave-lqkv"
server_key: "fd3dae6793ef6d9a07e5b40254172bdb"
server_key_iv: "e27f142e70d57f0ffbaa608aa431a14d"
server_key_version: "kuduclusterkey@0"
I20260929 19:03:06.255692 12852 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.001s
Sep 29 19:03:06 dist-test-slave-lqkv krb5kdc[12158](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790708586, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:03:06.271449 12869 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:06.271629 12852 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260929 19:03:06.271704 12852 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/minicluster-data/ts-1/wal
uuid: "6f7bc5348fe64b448f7da8c97840df38"
format_stamp: "Formatted at 2026-09-29 19:03:06 on dist-test-slave-lqkv"
server_key: "fd3dae6793ef6d9a07e5b40254172bdb"
server_key_iv: "e27f142e70d57f0ffbaa608aa431a14d"
server_key_version: "kuduclusterkey@0"
I20260929 19:03:06.271775 12852 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:03:06.291764 12852 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:03:06.294091 12852 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:03:06.294266 12852 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:03:06.294497 12852 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:03:06.294842 12852 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:03:06.294929 12852 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:06.294996 12852 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:03:06.295051 12852 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:06.300856 12852 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41345
I20260929 19:03:06.301251 12852 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/minicluster-data/ts-1/data/info.pb
I20260929 19:03:06.305382 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUbnLHE/build/release/bin/kudu as pid 12852
I20260929 19:03:06.305490 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/minicluster-data/ts-1/wal/instance
I20260929 19:03:06.305684 392 external_mini_cluster.cc:1428] Setting key d717844db9c547b02dcf9e287e3d01f1
I20260929 19:03:06.307221 12982 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41345 every 8 connection(s)
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Sep 29 19:03:06 dist-test-slave-lqkv krb5kdc[12158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708586, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260929 19:03:06.329880 12849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:06.308000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55681 (local address 127.0.98.62:37509)
0929 19:03:06.308043 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:06.308045 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:06.313179 (+ 5134us) server_negotiation.cc:407] Connection header received
0929 19:03:06.313800 (+ 621us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:06.313801 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:06.313813 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:06.313839 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:06.316689 (+ 2850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:06.317464 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:06.321793 (+ 4329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:06.322096 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:06.322155 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:06.326739 (+ 4584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:03:06.326741 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:03:06.326741 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:03:06.326748 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:03:06.328421 (+ 1673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:06.328858 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:06.328859 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:06.328859 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:06.328912 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:06.329167 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:06.329168 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:06.329168 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:06.329290 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:03:06.329336 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:06.329688 (+ 352us) server_negotiation.cc:299] Negotiation successful
0929 19:03:06.329774 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260929 19:03:06.330686 12986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:06.307743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37509 (local address 127.0.98.2:55681)
0929 19:03:06.313054 (+ 5311us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:06.313100 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:06.313687 (+ 587us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:06.314057 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:06.314062 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:06.314292 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:03:06.314567 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:06.314574 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:06.317583 (+ 3009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:06.317584 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:06.318772 (+ 1188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:06.318774 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:06.322254 (+ 3480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:06.322255 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:06.322391 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:06.323074 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:03:06.323083 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:03:06.326638 (+ 3555us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:03:06.328508 (+ 1870us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:03:06.328513 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:03:06.328515 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:03:06.328761 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:03:06.328972 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:03:06.328972 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:03:06.328973 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:03:06.329098 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:03:06.329363 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:03:06.329368 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:03:06.329538 (+ 170us) client_negotiation.cc:769] Sending connection context
0929 19:03:06.330240 (+ 702us) client_negotiation.cc:240] Negotiation successful
0929 19:03:06.330416 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5269,"thread_start_us":82,"threads_started":1}
I20260929 19:03:06.331323 12983 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37509
I20260929 19:03:06.331452 12983 heartbeater.cc:464] Registering TS with master...
I20260929 19:03:06.331619 12983 heartbeater.cc:511] Master 127.0.98.62:37509 requested a full tablet report, sending...
I20260929 19:03:06.331982 12634 ts_manager.cc:194] Registered new tserver with Master: 6f7bc5348fe64b448f7da8c97840df38 (127.0.98.2:41345)
I20260929 19:03:06.332345 12634 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:55681
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260929 19:03:06.343576 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUbnLHE/build/release/bin/kudu
/tmp/dist-test-taskUbnLHE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:56777/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37509
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:03:06.461874 12988 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:03:06.462106 12988 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:03:06.462136 12988 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:03:06.463671 12988 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:03:06.463722 12988 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:03:06.463757 12988 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 19:03:06.465920 12988 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:56777/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37509
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12988
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 18:09:13 UTC on ccycloud.quasar-hpkalf.root.comops.site
build id 4680913
I20260929 19:03:06.466144 12988 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:03:06.466421 12988 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:03:06.472203 12997 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:03:06.472174 12994 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:03:06.472174 12995 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:03:06.472878 12988 server_base.cc:1034] running on GCE node
I20260929 19:03:06.473085 12988 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:03:06.473321 12988 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:03:06.474498 12988 hybrid_clock.cc:648] HybridClock initialized: now 1790708586474473 us; error 38 us; skew 500 ppm
Sep 29 19:03:06 dist-test-slave-lqkv krb5kdc[12158](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708586, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:03:06.477604 12988 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 19:03:06.478071 12988 webserver.cc:460] Webserver started at http://127.0.98.3:44113/ using document root <none> and password file <none>
I20260929 19:03:06.478300 12988 fs_manager.cc:359] Metadata directory not provided
I20260929 19:03:06.478354 12988 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:03:06.478482 12988 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 19:03:06 dist-test-slave-lqkv krb5kdc[12158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708586, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260929 19:03:06.491086 12988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/minicluster-data/ts-2/data/instance:
uuid: "961680b75ba14cc6a393a9453f8d495a"
format_stamp: "Formatted at 2026-09-29 19:03:06 on dist-test-slave-lqkv"
server_key: "ad60c893ba81a0cadcc74c9c4f440204"
server_key_iv: "581558fc53de059f8899cc80073bbea2"
server_key_version: "kuduclusterkey@0"
I20260929 19:03:06.491552 12988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/minicluster-data/ts-2/wal/instance:
uuid: "961680b75ba14cc6a393a9453f8d495a"
format_stamp: "Formatted at 2026-09-29 19:03:06 on dist-test-slave-lqkv"
server_key: "ad60c893ba81a0cadcc74c9c4f440204"
server_key_iv: "581558fc53de059f8899cc80073bbea2"
server_key_version: "kuduclusterkey@0"
I20260929 19:03:06.514144 12988 fs_manager.cc:691] Time spent creating directory manager: real 0.022s user 0.001s sys 0.000s
I20260929 19:03:06.531376 13004 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:06.531615 12988 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260929 19:03:06.531703 12988 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/minicluster-data/ts-2/wal
uuid: "961680b75ba14cc6a393a9453f8d495a"
format_stamp: "Formatted at 2026-09-29 19:03:06 on dist-test-slave-lqkv"
server_key: "ad60c893ba81a0cadcc74c9c4f440204"
server_key_iv: "581558fc53de059f8899cc80073bbea2"
server_key_version: "kuduclusterkey@0"
I20260929 19:03:06.531783 12988 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:03:06.574662 12988 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:03:06.577106 12988 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:03:06.577382 12988 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:03:06.577656 12988 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:03:06.578053 12988 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:03:06.578145 12988 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:06.578217 12988 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:03:06.578274 12988 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:06.587247 12988 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33043
I20260929 19:03:06.587700 12988 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/minicluster-data/ts-2/data/info.pb
I20260929 19:03:06.589214 13117 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33043 every 8 connection(s)
I20260929 19:03:06.591969 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUbnLHE/build/release/bin/kudu as pid 12988
I20260929 19:03:06.592082 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/minicluster-data/ts-2/wal/instance
I20260929 19:03:06.592273 392 external_mini_cluster.cc:1428] Setting key 874ae2b990ab8ae0f6ed66b6656e282e
Sep 29 19:03:06 dist-test-slave-lqkv krb5kdc[12158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708586, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 19:03:06.599211 12849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:06.589073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50527 (local address 127.0.98.62:37509)
0929 19:03:06.589114 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:06.589128 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:06.589602 (+ 474us) server_negotiation.cc:407] Connection header received
0929 19:03:06.590292 (+ 690us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:06.590294 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:06.590306 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:06.590335 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:06.591120 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:06.591817 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:06.593385 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:06.593679 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:06.593720 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:06.596317 (+ 2597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:03:06.596318 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:03:06.596319 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:03:06.596325 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:03:06.597888 (+ 1563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:06.598289 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:06.598290 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:06.598290 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:06.598337 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:06.598562 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:06.598562 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:06.598563 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:06.598670 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:03:06.598705 (+ 35us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:06.599037 (+ 332us) server_negotiation.cc:299] Negotiation successful
0929 19:03:06.599125 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260929 19:03:06.599819 13120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:06.588784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37509 (local address 127.0.98.3:50527)
0929 19:03:06.589481 (+ 697us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:06.589521 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:06.590160 (+ 639us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:06.590509 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:06.590513 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:06.590746 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:03:06.591014 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:06.591020 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:06.591928 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:06.591929 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:06.593286 (+ 1357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:06.593289 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:06.593825 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:06.593826 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:06.593961 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:06.594527 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:03:06.594536 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:03:06.596219 (+ 1683us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:03:06.597971 (+ 1752us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:03:06.597976 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:03:06.597978 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:03:06.598214 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:03:06.598398 (+ 184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:03:06.598398 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:03:06.598398 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:03:06.598501 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:03:06.598741 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:03:06.598745 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:03:06.598906 (+ 161us) client_negotiation.cc:769] Sending connection context
0929 19:03:06.599496 (+ 590us) client_negotiation.cc:240] Negotiation successful
0929 19:03:06.599628 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":659,"thread_start_us":100,"threads_started":1}
I20260929 19:03:06.600400 13118 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37509
I20260929 19:03:06.600474 13118 heartbeater.cc:464] Registering TS with master...
I20260929 19:03:06.600620 13118 heartbeater.cc:511] Master 127.0.98.62:37509 requested a full tablet report, sending...
I20260929 19:03:06.600997 12634 ts_manager.cc:194] Registered new tserver with Master: 961680b75ba14cc6a393a9453f8d495a (127.0.98.3:33043)
I20260929 19:03:06.601465 12634 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:50527
I20260929 19:03:06.603694 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 19:03:06.604424 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUbnLHE/build/release/bin/kudu with pid 12715
I20260929 19:03:06.610546 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUbnLHE/build/release/bin/kudu with pid 12852
I20260929 19:03:06.616016 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUbnLHE/build/release/bin/kudu with pid 12988
I20260929 19:03:06.622226 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUbnLHE/build/release/bin/kudu with pid 12602
I20260929 19:03:06.627774 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUbnLHE/build/release/bin/kudu
/tmp/dist-test-taskUbnLHE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37509
--webserver_interface=127.0.98.62
--webserver_port=37231
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37509
--ranger_config_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:56777/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:56777/kms with env {KRB5CCNAME=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:03:06.702772 13121 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:03:06.702967 13121 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:03:06.702988 13121 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:03:06.704027 13121 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 19:03:06.704063 13121 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:03:06.704077 13121 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:03:06.704090 13121 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 19:03:06.704102 13121 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 19:03:06.705638 13121 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:56777/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37509
--ranger_config_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37509
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=37231
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13121
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 18:09:13 UTC on ccycloud.quasar-hpkalf.root.comops.site
build id 4680913
I20260929 19:03:06.705829 13121 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:03:06.705986 13121 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:03:06.710249 13129 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:03:06.710352 13127 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:03:06.710438 13121 server_base.cc:1034] running on GCE node
W20260929 19:03:06.710507 13126 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:03:06.710778 13121 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:03:06.711026 13121 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:03:06.712158 13121 hybrid_clock.cc:648] HybridClock initialized: now 1790708586712147 us; error 25 us; skew 500 ppm
Sep 29 19:03:06 dist-test-slave-lqkv krb5kdc[12158](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708586, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:03:06.714474 13121 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 19:03:06.714871 13121 webserver.cc:460] Webserver started at http://127.0.98.62:37231/ using document root <none> and password file <none>
I20260929 19:03:06.715087 13121 fs_manager.cc:359] Metadata directory not provided
I20260929 19:03:06.715157 13121 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 19:03:06 dist-test-slave-lqkv krb5kdc[12158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708586, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260929 19:03:06.770979 13121 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 19:03:06.771708 13137 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:06.771888 13121 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 19:03:06.771982 13121 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/minicluster-data/master-0/wal
uuid: "5ed01c77975c4dc98943ebf46fb3f54f"
format_stamp: "Formatted at 2026-09-29 19:03:05 on dist-test-slave-lqkv"
server_key: "5da362bfed92e1edd1f98cc9c4e75c2c"
server_key_iv: "043af3e2bf8b1cf6a9fe9987019a1302"
server_key_version: "kuduclusterkey@0"
I20260929 19:03:06.772318 13121 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:03:06.783185 13121 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:03:06.785162 13121 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:03:06.785300 13121 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:03:06.789177 13121 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37509
I20260929 19:03:06.789206 13189 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37509 every 8 connection(s)
I20260929 19:03:06.789556 13121 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/minicluster-data/master-0/data/info.pb
I20260929 19:03:06.790663 13190 sys_catalog.cc:263] Verifying existing consensus state
I20260929 19:03:06.791165 13190 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ed01c77975c4dc98943ebf46fb3f54f: Bootstrap starting.
I20260929 19:03:06.792856 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUbnLHE/build/release/bin/kudu as pid 13121
I20260929 19:03:06.793195 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUbnLHE/build/release/bin/kudu
/tmp/dist-test-taskUbnLHE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:56777/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:44183
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:37509
--webserver_port=33915
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:56777/kms with env {KRB5CCNAME=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 19:03:06.799111 13190 log.cc:824] T 00000000000000000000000000000000 P 5ed01c77975c4dc98943ebf46fb3f54f: Log is configured to *not* fsync() on all Append() calls
I20260929 19:03:06.800655 13190 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ed01c77975c4dc98943ebf46fb3f54f: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 19:03:06.800940 13190 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ed01c77975c4dc98943ebf46fb3f54f: Bootstrap complete.
I20260929 19:03:06.803001 13190 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5ed01c77975c4dc98943ebf46fb3f54f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ed01c77975c4dc98943ebf46fb3f54f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37509 } }
I20260929 19:03:06.803193 13190 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5ed01c77975c4dc98943ebf46fb3f54f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ed01c77975c4dc98943ebf46fb3f54f, State: Initialized, Role: FOLLOWER
I20260929 19:03:06.803331 13190 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5ed01c77975c4dc98943ebf46fb3f54f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ed01c77975c4dc98943ebf46fb3f54f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37509 } }
I20260929 19:03:06.803391 13190 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5ed01c77975c4dc98943ebf46fb3f54f [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 19:03:06.803438 13190 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5ed01c77975c4dc98943ebf46fb3f54f [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 19:03:06.803483 13190 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5ed01c77975c4dc98943ebf46fb3f54f [term 1 FOLLOWER]: Advancing to term 2
I20260929 19:03:06.804445 13190 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5ed01c77975c4dc98943ebf46fb3f54f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ed01c77975c4dc98943ebf46fb3f54f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37509 } }
I20260929 19:03:06.804550 13190 leader_election.cc:302] T 00000000000000000000000000000000 P 5ed01c77975c4dc98943ebf46fb3f54f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5ed01c77975c4dc98943ebf46fb3f54f; no voters:
I20260929 19:03:06.804723 13190 leader_election.cc:288] T 00000000000000000000000000000000 P 5ed01c77975c4dc98943ebf46fb3f54f [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 19:03:06.804756 13194 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5ed01c77975c4dc98943ebf46fb3f54f [term 2 FOLLOWER]: Leader election won for term 2
I20260929 19:03:06.804937 13194 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5ed01c77975c4dc98943ebf46fb3f54f [term 2 LEADER]: Becoming Leader. State: Replica: 5ed01c77975c4dc98943ebf46fb3f54f, State: Running, Role: LEADER
I20260929 19:03:06.804978 13190 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5ed01c77975c4dc98943ebf46fb3f54f [sys.catalog]: configured and running, proceeding with master startup.
I20260929 19:03:06.805087 13194 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5ed01c77975c4dc98943ebf46fb3f54f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ed01c77975c4dc98943ebf46fb3f54f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37509 } }
I20260929 19:03:06.805480 13196 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ed01c77975c4dc98943ebf46fb3f54f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5ed01c77975c4dc98943ebf46fb3f54f. Latest consensus state: current_term: 2 leader_uuid: "5ed01c77975c4dc98943ebf46fb3f54f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ed01c77975c4dc98943ebf46fb3f54f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37509 } } }
I20260929 19:03:06.805549 13196 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ed01c77975c4dc98943ebf46fb3f54f [sys.catalog]: This master's current role is: LEADER
I20260929 19:03:06.805718 13195 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ed01c77975c4dc98943ebf46fb3f54f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "5ed01c77975c4dc98943ebf46fb3f54f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ed01c77975c4dc98943ebf46fb3f54f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37509 } } }
I20260929 19:03:06.805800 13195 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ed01c77975c4dc98943ebf46fb3f54f [sys.catalog]: This master's current role is: LEADER
I20260929 19:03:06.805918 13200 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 19:03:06.806310 13200 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 19:03:06.806474 13200 catalog_manager.cc:1259] Loaded cluster ID: 8552eba881a545f2a4f236f5b2bffa0f
I20260929 19:03:06.806536 13200 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 19:03:06.806627 13190 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260929 19:03:06.807751 13200 catalog_manager.cc:1504] Loading token signing keys...
I20260929 19:03:06.807902 13200 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 5ed01c77975c4dc98943ebf46fb3f54f: Loaded TSK: 0
I20260929 19:03:06.808085 13200 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 19:03:06.871083 13192 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:03:06.871263 13192 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:03:06.871297 13192 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:03:06.872516 13192 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:03:06.872581 13192 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:03:06.872619 13192 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 19:03:06.874691 13192 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:56777/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:44183
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=33915
--tserver_master_addrs=127.0.98.62:37509
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13192
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 18:09:13 UTC on ccycloud.quasar-hpkalf.root.comops.site
build id 4680913
I20260929 19:03:06.875156 13192 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:03:06.875445 13192 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:03:06.880963 13227 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:03:06.881026 13230 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:03:06.880959 13228 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:03:06.881428 13192 server_base.cc:1034] running on GCE node
I20260929 19:03:06.881652 13192 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:03:06.881944 13192 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:03:06.883121 13192 hybrid_clock.cc:648] HybridClock initialized: now 1790708586883101 us; error 31 us; skew 500 ppm
Sep 29 19:03:06 dist-test-slave-lqkv krb5kdc[12158](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708586, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:03:06.886258 13192 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 19:03:06.886804 13192 webserver.cc:460] Webserver started at http://127.0.98.1:33915/ using document root <none> and password file <none>
I20260929 19:03:06.887132 13192 fs_manager.cc:359] Metadata directory not provided
I20260929 19:03:06.887269 13192 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 19:03:06 dist-test-slave-lqkv krb5kdc[12158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708586, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260929 19:03:06.909436 13192 fs_manager.cc:709] Time spent opening directory manager: real 0.007s user 0.001s sys 0.000s
I20260929 19:03:06.912349 13238 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:06.912549 13192 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 19:03:06.912654 13192 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/minicluster-data/ts-0/wal
uuid: "e5dca27a04a448bb82a8bcaeeae3f252"
format_stamp: "Formatted at 2026-09-29 19:03:06 on dist-test-slave-lqkv"
server_key: "ae0c119149d45e66ff47b2ae47374f0e"
server_key_iv: "75b4b306e1d823baa45e7d2e23ba8dcf"
server_key_version: "kuduclusterkey@0"
I20260929 19:03:06.913096 13192 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:03:06.930452 13192 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:03:06.932596 13192 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:03:06.932780 13192 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:03:06.933024 13192 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:03:06.933418 13192 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:03:06.933501 13192 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:06.933609 13192 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:03:06.933698 13192 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:06.939986 13192 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44183
I20260929 19:03:06.940212 13351 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44183 every 8 connection(s)
I20260929 19:03:06.940474 13192 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/minicluster-data/ts-0/data/info.pb
I20260929 19:03:06.948671 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUbnLHE/build/release/bin/kudu as pid 13192
Sep 29 19:03:06 dist-test-slave-lqkv krb5kdc[12158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708586, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 19:03:06.949030 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUbnLHE/build/release/bin/kudu
/tmp/dist-test-taskUbnLHE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:56777/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:41345
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:37509
--webserver_port=37787
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:56777/kms with env {KRB5CCNAME=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 19:03:06.954423 13354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:06.941102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37509 (local address 127.0.98.1:53753)
0929 19:03:06.941563 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:06.941607 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:06.942210 (+ 603us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:06.942758 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:06.942762 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:06.942964 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:03:06.943233 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:06.943239 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:06.945008 (+ 1769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:06.945009 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:06.946216 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:06.946219 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:06.946979 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:06.946980 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:06.947108 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:06.947670 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:03:06.947679 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:03:06.950844 (+ 3165us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:03:06.952787 (+ 1943us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:03:06.952791 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:03:06.952793 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:03:06.953053 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:03:06.953282 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:03:06.953283 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:03:06.953283 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:03:06.953380 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:03:06.953791 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:03:06.953796 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:03:06.953929 (+ 133us) client_negotiation.cc:769] Sending connection context
0929 19:03:06.954044 (+ 115us) client_negotiation.cc:240] Negotiation successful
0929 19:03:06.954207 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":73,"threads_started":1}
I20260929 19:03:06.955361 13355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:06.941466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53753 (local address 127.0.98.62:37509)
0929 19:03:06.942280 (+ 814us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:06.942289 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:06.942327 (+ 38us) server_negotiation.cc:407] Connection header received
0929 19:03:06.942494 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:06.942501 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:06.942564 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:06.942699 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:06.943365 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:06.944895 (+ 1530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:06.946312 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:06.946858 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:06.946968 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:06.950973 (+ 4005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:03:06.950977 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:03:06.950978 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:03:06.950988 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:03:06.952693 (+ 1705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:06.953134 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:06.953137 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:06.953137 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:06.953223 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:06.953435 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:06.953435 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:06.953435 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:06.953691 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:03:06.953764 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:06.954996 (+ 1232us) server_negotiation.cc:299] Negotiation successful
0929 19:03:06.955112 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":763,"thread_start_us":101,"threads_started":1}
I20260929 19:03:06.956207 13352 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37509
I20260929 19:03:06.956290 13352 heartbeater.cc:464] Registering TS with master...
I20260929 19:03:06.956449 13352 heartbeater.cc:511] Master 127.0.98.62:37509 requested a full tablet report, sending...
W20260929 19:03:06.957304 13352 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:37509 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260929 19:03:06.957368 13352 heartbeater.cc:464] Registering TS with master...
I20260929 19:03:06.957422 13352 heartbeater.cc:511] Master 127.0.98.62:37509 requested a full tablet report, sending...
I20260929 19:03:06.957813 13352 heartbeater.cc:464] Registering TS with master...
I20260929 19:03:06.957880 13352 heartbeater.cc:511] Master 127.0.98.62:37509 requested a full tablet report, sending...
W20260929 19:03:06.958210 13352 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 19:03:07.049018 13356 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:03:07.049340 13356 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:03:07.049436 13356 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:03:07.051190 13356 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:03:07.051311 13356 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:03:07.051407 13356 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 19:03:07.053475 13356 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:56777/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:41345
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=37787
--tserver_master_addrs=127.0.98.62:37509
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13356
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 18:09:13 UTC on ccycloud.quasar-hpkalf.root.comops.site
build id 4680913
I20260929 19:03:07.053838 13356 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:03:07.054126 13356 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:03:07.060338 13364 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:03:07.060420 13361 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:03:07.060498 13362 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:03:07.060685 13356 server_base.cc:1034] running on GCE node
I20260929 19:03:07.060940 13356 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:03:07.061187 13356 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:03:07.062342 13356 hybrid_clock.cc:648] HybridClock initialized: now 1790708587062324 us; error 33 us; skew 500 ppm
Sep 29 19:03:07 dist-test-slave-lqkv krb5kdc[12158](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708587, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:03:07.065157 13356 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 19:03:07.065675 13356 webserver.cc:460] Webserver started at http://127.0.98.2:37787/ using document root <none> and password file <none>
I20260929 19:03:07.065893 13356 fs_manager.cc:359] Metadata directory not provided
I20260929 19:03:07.065979 13356 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 19:03:07 dist-test-slave-lqkv krb5kdc[12158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708587, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260929 19:03:07.088028 13356 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.000s
I20260929 19:03:07.089159 13372 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:07.089311 13356 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 19:03:07.089391 13356 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/minicluster-data/ts-1/wal
uuid: "6f7bc5348fe64b448f7da8c97840df38"
format_stamp: "Formatted at 2026-09-29 19:03:06 on dist-test-slave-lqkv"
server_key: "fd3dae6793ef6d9a07e5b40254172bdb"
server_key_iv: "e27f142e70d57f0ffbaa608aa431a14d"
server_key_version: "kuduclusterkey@0"
I20260929 19:03:07.089748 13356 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:03:07.108579 13356 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:03:07.110502 13356 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:03:07.110646 13356 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:03:07.110850 13356 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:03:07.111193 13356 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:03:07.111275 13356 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:07.111330 13356 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:03:07.111374 13356 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:07.118402 13356 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41345
I20260929 19:03:07.118471 13485 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41345 every 8 connection(s)
I20260929 19:03:07.118822 13356 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/minicluster-data/ts-1/data/info.pb
Sep 29 19:03:07 dist-test-slave-lqkv krb5kdc[12158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708587, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 19:03:07.125138 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUbnLHE/build/release/bin/kudu as pid 13356
I20260929 19:03:07.125447 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUbnLHE/build/release/bin/kudu
/tmp/dist-test-taskUbnLHE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:56777/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:33043
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:37509
--webserver_port=44113
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:56777/kms with env {KRB5CCNAME=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 19:03:07.128521 13355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:07.119494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38959 (local address 127.0.98.62:37509)
0929 19:03:07.119559 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:07.119561 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:07.119778 (+ 217us) server_negotiation.cc:407] Connection header received
0929 19:03:07.120334 (+ 556us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:07.120335 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:07.120346 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:07.120441 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:07.121095 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:07.121808 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:07.122798 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:07.123072 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:07.123141 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:07.125358 (+ 2217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:03:07.125360 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:03:07.125360 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:03:07.125367 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:03:07.127028 (+ 1661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:07.127505 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:07.127507 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:07.127507 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:07.127567 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:07.127843 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:07.127844 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:07.127844 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:07.127982 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:03:07.128021 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:07.128345 (+ 324us) server_negotiation.cc:299] Negotiation successful
0929 19:03:07.128428 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260929 19:03:07.128635 13488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:07.119482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37509 (local address 127.0.98.2:38959)
0929 19:03:07.119638 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:07.119674 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:07.120217 (+ 543us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:07.120524 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:07.120528 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:07.120735 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:03:07.120993 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:07.120999 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:07.121901 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:07.121902 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:07.122709 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:07.122711 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:07.123191 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:07.123192 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:07.123276 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:07.123715 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:03:07.123724 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:03:07.125252 (+ 1528us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:03:07.127122 (+ 1870us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:03:07.127128 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:03:07.127130 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:03:07.127424 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:03:07.127649 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:03:07.127650 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:03:07.127650 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:03:07.127773 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:03:07.128065 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:03:07.128069 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:03:07.128233 (+ 164us) client_negotiation.cc:769] Sending connection context
0929 19:03:07.128326 (+ 93us) client_negotiation.cc:240] Negotiation successful
0929 19:03:07.128444 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":101,"threads_started":1}
I20260929 19:03:07.133273 13486 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37509
I20260929 19:03:07.133363 13486 heartbeater.cc:464] Registering TS with master...
I20260929 19:03:07.133540 13486 heartbeater.cc:511] Master 127.0.98.62:37509 requested a full tablet report, sending...
W20260929 19:03:07.134232 13486 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:37509 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260929 19:03:07.134300 13486 heartbeater.cc:464] Registering TS with master...
I20260929 19:03:07.134361 13486 heartbeater.cc:511] Master 127.0.98.62:37509 requested a full tablet report, sending...
I20260929 19:03:07.134832 13486 heartbeater.cc:464] Registering TS with master...
I20260929 19:03:07.134923 13486 heartbeater.cc:511] Master 127.0.98.62:37509 requested a full tablet report, sending...
W20260929 19:03:07.135439 13486 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 19:03:07.216075 13489 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:03:07.216415 13489 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:03:07.216514 13489 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:03:07.218743 13489 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:03:07.218807 13489 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:03:07.218881 13489 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 19:03:07.221017 13489 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:56777/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:33043
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=44113
--tserver_master_addrs=127.0.98.62:37509
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13489
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 18:09:13 UTC on ccycloud.quasar-hpkalf.root.comops.site
build id 4680913
I20260929 19:03:07.221388 13489 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:03:07.221625 13489 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:03:07.229816 13497 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:03:07.229846 13495 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:03:07.230405 13494 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:03:07.230777 13489 server_base.cc:1034] running on GCE node
I20260929 19:03:07.230985 13489 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:03:07.231216 13489 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:03:07.232376 13489 hybrid_clock.cc:648] HybridClock initialized: now 1790708587232338 us; error 46 us; skew 500 ppm
Sep 29 19:03:07 dist-test-slave-lqkv krb5kdc[12158](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708587, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:03:07.235639 13489 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 19:03:07.236114 13489 webserver.cc:460] Webserver started at http://127.0.98.3:44113/ using document root <none> and password file <none>
I20260929 19:03:07.236333 13489 fs_manager.cc:359] Metadata directory not provided
I20260929 19:03:07.236394 13489 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 19:03:07 dist-test-slave-lqkv krb5kdc[12158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708587, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260929 19:03:07.266355 13489 fs_manager.cc:709] Time spent opening directory manager: real 0.015s user 0.001s sys 0.000s
I20260929 19:03:07.276906 13505 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:07.277139 13489 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.000s sys 0.000s
I20260929 19:03:07.277236 13489 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/minicluster-data/ts-2/wal
uuid: "961680b75ba14cc6a393a9453f8d495a"
format_stamp: "Formatted at 2026-09-29 19:03:06 on dist-test-slave-lqkv"
server_key: "ad60c893ba81a0cadcc74c9c4f440204"
server_key_iv: "581558fc53de059f8899cc80073bbea2"
server_key_version: "kuduclusterkey@0"
I20260929 19:03:07.277685 13489 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:03:07.342311 13489 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:03:07.344228 13489 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:03:07.344394 13489 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:03:07.344631 13489 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:03:07.345068 13489 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:03:07.345151 13489 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:07.345216 13489 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:03:07.345280 13489 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:07.352377 13489 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33043
I20260929 19:03:07.352881 13489 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/minicluster-data/ts-2/data/info.pb
I20260929 19:03:07.355480 13619 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33043 every 8 connection(s)
Sep 29 19:03:07 dist-test-slave-lqkv krb5kdc[12158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708587, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 19:03:07.363253 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUbnLHE/build/release/bin/kudu as pid 13489
I20260929 19:03:07.365729 13622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:07.356070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37509 (local address 127.0.98.3:38991)
0929 19:03:07.356233 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:07.356279 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:07.356931 (+ 652us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:07.357304 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:07.357308 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:07.357546 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:03:07.357807 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:07.357813 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:07.358613 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:07.358614 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:07.359391 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:07.359394 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:07.359875 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:07.359876 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:07.359979 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:07.360456 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:03:07.360463 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:03:07.361967 (+ 1504us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:03:07.364008 (+ 2041us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:03:07.364014 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:03:07.364016 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:03:07.364353 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:03:07.364565 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:03:07.364566 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:03:07.364566 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:03:07.364707 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:03:07.365064 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:03:07.365070 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:03:07.365248 (+ 178us) client_negotiation.cc:769] Sending connection context
0929 19:03:07.365365 (+ 117us) client_negotiation.cc:240] Negotiation successful
0929 19:03:07.365517 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":84,"threads_started":1}
I20260929 19:03:07.366387 13355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:07.356051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38991 (local address 127.0.98.62:37509)
0929 19:03:07.356099 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:07.356101 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:07.356365 (+ 264us) server_negotiation.cc:407] Connection header received
0929 19:03:07.357074 (+ 709us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:07.357076 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:07.357090 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:07.357171 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:07.357915 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:07.358510 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:07.359489 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:07.359777 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:07.359856 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:07.362063 (+ 2207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:03:07.362065 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:03:07.362065 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:03:07.362072 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:03:07.363876 (+ 1804us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:07.364431 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:07.364432 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:07.364432 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:07.364503 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:07.364795 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:07.364796 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:07.364796 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:07.364972 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:03:07.365036 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:07.366217 (+ 1181us) server_negotiation.cc:299] Negotiation successful
0929 19:03:07.366298 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260929 19:03:07.367110 13620 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37509
I20260929 19:03:07.367199 13620 heartbeater.cc:464] Registering TS with master...
I20260929 19:03:07.367373 13620 heartbeater.cc:511] Master 127.0.98.62:37509 requested a full tablet report, sending...
W20260929 19:03:07.368010 13620 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:37509 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260929 19:03:07.368078 13620 heartbeater.cc:464] Registering TS with master...
I20260929 19:03:07.368139 13620 heartbeater.cc:511] Master 127.0.98.62:37509 requested a full tablet report, sending...
I20260929 19:03:07.368515 13620 heartbeater.cc:464] Registering TS with master...
I20260929 19:03:07.368588 13620 heartbeater.cc:511] Master 127.0.98.62:37509 requested a full tablet report, sending...
W20260929 19:03:07.368947 13620 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 19:03:07.375619 13624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:07.364038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40286 (local address 127.0.98.62:37509)
0929 19:03:07.364185 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:07.364187 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:07.364195 (+ 8us) server_negotiation.cc:407] Connection header received
0929 19:03:07.364215 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:07.364216 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:07.364228 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:07.364280 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:07.365359 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:07.365939 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:07.368778 (+ 2839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:07.369115 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:07.369209 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:07.370454 (+ 1245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:03:07.370455 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:03:07.370456 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:03:07.370461 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:03:07.373466 (+ 3005us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:07.374760 (+ 1294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:07.374761 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:07.374761 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:07.374825 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:07.375040 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:07.375040 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:07.375041 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:07.375170 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:03:07.375383 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:07.375424 (+ 41us) server_negotiation.cc:299] Negotiation successful
0929 19:03:07.375503 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":65,"threads_started":1}
Sep 29 19:03:07 dist-test-slave-lqkv krb5kdc[12158](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790708587, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:03:07.958402 13352 heartbeater.cc:464] Registering TS with master...
I20260929 19:03:07.958549 13352 heartbeater.cc:511] Master 127.0.98.62:37509 requested a full tablet report, sending...
I20260929 19:03:07.958935 13154 ts_manager.cc:194] Registered new tserver with Master: e5dca27a04a448bb82a8bcaeeae3f252 (127.0.98.1:44183)
I20260929 19:03:07.959900 13154 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:53753
I20260929 19:03:08.135622 13486 heartbeater.cc:464] Registering TS with master...
I20260929 19:03:08.135746 13486 heartbeater.cc:511] Master 127.0.98.62:37509 requested a full tablet report, sending...
I20260929 19:03:08.136001 13154 ts_manager.cc:194] Registered new tserver with Master: 6f7bc5348fe64b448f7da8c97840df38 (127.0.98.2:41345)
I20260929 19:03:08.136557 13154 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:38959
Sep 29 19:03:08 dist-test-slave-lqkv krb5kdc[12158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708587, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 29 19:03:08 dist-test-slave-lqkv krb5kdc[12158](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708587, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260929 19:03:08.369091 13620 heartbeater.cc:464] Registering TS with master...
I20260929 19:03:08.369208 13620 heartbeater.cc:511] Master 127.0.98.62:37509 requested a full tablet report, sending...
I20260929 19:03:08.369467 13154 ts_manager.cc:194] Registered new tserver with Master: 961680b75ba14cc6a393a9453f8d495a (127.0.98.3:33043)
I20260929 19:03:08.369903 13154 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:38991
I20260929 19:03:08.373337 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 19:03:08.393213 13643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:08.375075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40288 (local address 127.0.98.62:37509)
0929 19:03:08.384972 (+ 9897us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:08.384974 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:08.384984 (+ 10us) server_negotiation.cc:407] Connection header received
0929 19:03:08.385008 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:08.385009 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:08.385020 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:08.385092 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:08.385895 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:08.386675 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:08.387666 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:08.387940 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:08.387993 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:08.389728 (+ 1735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:03:08.389729 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:03:08.389729 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:03:08.389736 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:03:08.391791 (+ 2055us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:08.392170 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:08.392172 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:08.392172 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:08.392228 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:08.392413 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:08.392414 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:08.392414 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:08.392544 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:03:08.392630 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:08.393046 (+ 416us) server_negotiation.cc:299] Negotiation successful
0929 19:03:08.393119 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9872,"thread_start_us":71,"threads_started":1}
I20260929 19:03:08.395298 13154 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40288:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 19:03:08.397594 13154 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 19:03:08.416716 13652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:08.406775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33043 (local address 127.0.0.1:33552)
0929 19:03:08.407121 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:08.407157 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:08.407215 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:08.413126 (+ 5911us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:08.413128 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:08.413131 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:03:08.413198 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:08.413199 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:08.414314 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:08.414315 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:08.415536 (+ 1221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:08.415539 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:08.416441 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:08.416442 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:08.416525 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:08.416536 (+ 11us) client_negotiation.cc:769] Sending connection context
0929 19:03:08.416596 (+ 60us) client_negotiation.cc:240] Negotiation successful
0929 19:03:08.416640 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":59,"threads_started":1}
I20260929 19:03:08.416985 13654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:08.407156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33552 (local address 127.0.98.3:33043)
0929 19:03:08.412888 (+ 5732us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:08.412892 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:08.412902 (+ 10us) server_negotiation.cc:407] Connection header received
0929 19:03:08.412936 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:08.412938 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:08.412964 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:08.413037 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:03:08.413274 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:08.414189 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:08.415614 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:08.416285 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:08.416347 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:08.416411 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:08.416880 (+ 469us) server_negotiation.cc:299] Negotiation successful
0929 19:03:08.416917 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5695,"thread_start_us":47,"threads_started":1}
I20260929 19:03:08.417907 13554 tablet_service.cc:1467] Processing CreateTablet for tablet accba90b768d4e7e907312f7837ef0c4 (DEFAULT_TABLE table=test-table [id=62cfa5512c4a4c7291a7ea703f4303db]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:03:08.418280 13554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet accba90b768d4e7e907312f7837ef0c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:03:08.421010 13652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:08.406974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41345 (local address 127.0.0.1:50514)
0929 19:03:08.417505 (+ 10531us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:08.417515 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:08.417555 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:08.417792 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:08.417793 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:08.417794 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:03:08.417818 (+ 24us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:08.417819 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:08.418878 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:08.418879 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:08.419815 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:08.419817 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:08.420782 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:08.420783 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:08.420856 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:08.420859 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 19:03:08.420907 (+ 48us) client_negotiation.cc:240] Negotiation successful
0929 19:03:08.420940 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10522,"thread_start_us":61,"threads_started":1}
I20260929 19:03:08.421288 13653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:08.407051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50514 (local address 127.0.98.2:41345)
0929 19:03:08.413026 (+ 5975us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:08.413031 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:08.417611 (+ 4580us) server_negotiation.cc:407] Connection header received
0929 19:03:08.417648 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:08.417650 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:08.417664 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:08.417721 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:03:08.417881 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:08.418791 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:08.419905 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:08.420617 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:08.420685 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:08.420751 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:08.421163 (+ 412us) server_negotiation.cc:299] Negotiation successful
0929 19:03:08.421209 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5940,"thread_start_us":52,"threads_started":1}
I20260929 19:03:08.421875 13420 tablet_service.cc:1467] Processing CreateTablet for tablet accba90b768d4e7e907312f7837ef0c4 (DEFAULT_TABLE table=test-table [id=62cfa5512c4a4c7291a7ea703f4303db]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:03:08.422159 13420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet accba90b768d4e7e907312f7837ef0c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:03:08.425568 13655 tablet_bootstrap.cc:492] T accba90b768d4e7e907312f7837ef0c4 P 961680b75ba14cc6a393a9453f8d495a: Bootstrap starting.
I20260929 19:03:08.426219 13655 tablet_bootstrap.cc:654] T accba90b768d4e7e907312f7837ef0c4 P 961680b75ba14cc6a393a9453f8d495a: Neither blocks nor log segments found. Creating new log.
I20260929 19:03:08.426399 13656 tablet_bootstrap.cc:492] T accba90b768d4e7e907312f7837ef0c4 P 6f7bc5348fe64b448f7da8c97840df38: Bootstrap starting.
I20260929 19:03:08.427547 13656 tablet_bootstrap.cc:654] T accba90b768d4e7e907312f7837ef0c4 P 6f7bc5348fe64b448f7da8c97840df38: Neither blocks nor log segments found. Creating new log.
I20260929 19:03:08.427925 13655 log.cc:824] T accba90b768d4e7e907312f7837ef0c4 P 961680b75ba14cc6a393a9453f8d495a: Log is configured to *not* fsync() on all Append() calls
I20260929 19:03:08.428360 13656 log.cc:824] T accba90b768d4e7e907312f7837ef0c4 P 6f7bc5348fe64b448f7da8c97840df38: Log is configured to *not* fsync() on all Append() calls
I20260929 19:03:08.428973 13649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:08.406812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44183 (local address 127.0.0.1:59950)
0929 19:03:08.415657 (+ 8845us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:08.415666 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:08.415706 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:08.415960 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:08.415961 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:08.415963 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:03:08.415995 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:08.415996 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:08.417326 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:08.417327 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:08.422914 (+ 5587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:08.422916 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:08.428675 (+ 5759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:08.428676 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:08.428742 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:08.428746 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 19:03:08.428796 (+ 50us) client_negotiation.cc:240] Negotiation successful
0929 19:03:08.428859 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8829,"thread_start_us":77,"threads_started":1}
I20260929 19:03:08.429281 13656 tablet_bootstrap.cc:492] T accba90b768d4e7e907312f7837ef0c4 P 6f7bc5348fe64b448f7da8c97840df38: No bootstrap required, opened a new log
I20260929 19:03:08.429346 13656 ts_tablet_manager.cc:1397] T accba90b768d4e7e907312f7837ef0c4 P 6f7bc5348fe64b448f7da8c97840df38: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260929 19:03:08.429450 13651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:08.406904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59950 (local address 127.0.98.1:44183)
0929 19:03:08.412940 (+ 6036us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:08.412945 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:08.415758 (+ 2813us) server_negotiation.cc:407] Connection header received
0929 19:03:08.415795 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:08.415798 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:08.415812 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:08.415877 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:03:08.416059 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:08.417235 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:08.427689 (+ 10454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:08.428517 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:08.428586 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:08.428642 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:08.429323 (+ 681us) server_negotiation.cc:299] Negotiation successful
0929 19:03:08.429372 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5993,"thread_start_us":48,"threads_started":1}
I20260929 19:03:08.430013 13286 tablet_service.cc:1467] Processing CreateTablet for tablet accba90b768d4e7e907312f7837ef0c4 (DEFAULT_TABLE table=test-table [id=62cfa5512c4a4c7291a7ea703f4303db]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:03:08.430306 13286 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet accba90b768d4e7e907312f7837ef0c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:03:08.431782 13656 raft_consensus.cc:348] T accba90b768d4e7e907312f7837ef0c4 P 6f7bc5348fe64b448f7da8c97840df38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "961680b75ba14cc6a393a9453f8d495a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33043 } } peers { permanent_uuid: "e5dca27a04a448bb82a8bcaeeae3f252" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44183 } } peers { permanent_uuid: "6f7bc5348fe64b448f7da8c97840df38" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41345 } }
I20260929 19:03:08.431928 13656 raft_consensus.cc:374] T accba90b768d4e7e907312f7837ef0c4 P 6f7bc5348fe64b448f7da8c97840df38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:03:08.431957 13656 raft_consensus.cc:729] T accba90b768d4e7e907312f7837ef0c4 P 6f7bc5348fe64b448f7da8c97840df38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f7bc5348fe64b448f7da8c97840df38, State: Initialized, Role: FOLLOWER
I20260929 19:03:08.433121 13656 consensus_queue.cc:260] T accba90b768d4e7e907312f7837ef0c4 P 6f7bc5348fe64b448f7da8c97840df38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "961680b75ba14cc6a393a9453f8d495a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33043 } } peers { permanent_uuid: "e5dca27a04a448bb82a8bcaeeae3f252" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44183 } } peers { permanent_uuid: "6f7bc5348fe64b448f7da8c97840df38" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41345 } }
I20260929 19:03:08.433467 13656 ts_tablet_manager.cc:1428] T accba90b768d4e7e907312f7837ef0c4 P 6f7bc5348fe64b448f7da8c97840df38: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260929 19:03:08.433612 13486 heartbeater.cc:503] Master 127.0.98.62:37509 was elected leader, sending a full tablet report...
I20260929 19:03:08.440966 13659 tablet_bootstrap.cc:492] T accba90b768d4e7e907312f7837ef0c4 P e5dca27a04a448bb82a8bcaeeae3f252: Bootstrap starting.
I20260929 19:03:08.441538 13659 tablet_bootstrap.cc:654] T accba90b768d4e7e907312f7837ef0c4 P e5dca27a04a448bb82a8bcaeeae3f252: Neither blocks nor log segments found. Creating new log.
I20260929 19:03:08.441866 13659 log.cc:824] T accba90b768d4e7e907312f7837ef0c4 P e5dca27a04a448bb82a8bcaeeae3f252: Log is configured to *not* fsync() on all Append() calls
I20260929 19:03:08.442512 13659 tablet_bootstrap.cc:492] T accba90b768d4e7e907312f7837ef0c4 P e5dca27a04a448bb82a8bcaeeae3f252: No bootstrap required, opened a new log
I20260929 19:03:08.442574 13659 ts_tablet_manager.cc:1397] T accba90b768d4e7e907312f7837ef0c4 P e5dca27a04a448bb82a8bcaeeae3f252: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 19:03:08.444893 13659 raft_consensus.cc:348] T accba90b768d4e7e907312f7837ef0c4 P e5dca27a04a448bb82a8bcaeeae3f252 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "961680b75ba14cc6a393a9453f8d495a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33043 } } peers { permanent_uuid: "e5dca27a04a448bb82a8bcaeeae3f252" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44183 } } peers { permanent_uuid: "6f7bc5348fe64b448f7da8c97840df38" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41345 } }
I20260929 19:03:08.445034 13659 raft_consensus.cc:374] T accba90b768d4e7e907312f7837ef0c4 P e5dca27a04a448bb82a8bcaeeae3f252 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:03:08.445065 13659 raft_consensus.cc:729] T accba90b768d4e7e907312f7837ef0c4 P e5dca27a04a448bb82a8bcaeeae3f252 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5dca27a04a448bb82a8bcaeeae3f252, State: Initialized, Role: FOLLOWER
I20260929 19:03:08.445169 13659 consensus_queue.cc:260] T accba90b768d4e7e907312f7837ef0c4 P e5dca27a04a448bb82a8bcaeeae3f252 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "961680b75ba14cc6a393a9453f8d495a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33043 } } peers { permanent_uuid: "e5dca27a04a448bb82a8bcaeeae3f252" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44183 } } peers { permanent_uuid: "6f7bc5348fe64b448f7da8c97840df38" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41345 } }
I20260929 19:03:08.445838 13659 ts_tablet_manager.cc:1428] T accba90b768d4e7e907312f7837ef0c4 P e5dca27a04a448bb82a8bcaeeae3f252: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 19:03:08.446048 13352 heartbeater.cc:503] Master 127.0.98.62:37509 was elected leader, sending a full tablet report...
I20260929 19:03:08.449165 13655 tablet_bootstrap.cc:492] T accba90b768d4e7e907312f7837ef0c4 P 961680b75ba14cc6a393a9453f8d495a: No bootstrap required, opened a new log
I20260929 19:03:08.449236 13655 ts_tablet_manager.cc:1397] T accba90b768d4e7e907312f7837ef0c4 P 961680b75ba14cc6a393a9453f8d495a: Time spent bootstrapping tablet: real 0.024s user 0.002s sys 0.000s
I20260929 19:03:08.451515 13655 raft_consensus.cc:348] T accba90b768d4e7e907312f7837ef0c4 P 961680b75ba14cc6a393a9453f8d495a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "961680b75ba14cc6a393a9453f8d495a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33043 } } peers { permanent_uuid: "e5dca27a04a448bb82a8bcaeeae3f252" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44183 } } peers { permanent_uuid: "6f7bc5348fe64b448f7da8c97840df38" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41345 } }
I20260929 19:03:08.451663 13655 raft_consensus.cc:374] T accba90b768d4e7e907312f7837ef0c4 P 961680b75ba14cc6a393a9453f8d495a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:03:08.451689 13655 raft_consensus.cc:729] T accba90b768d4e7e907312f7837ef0c4 P 961680b75ba14cc6a393a9453f8d495a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 961680b75ba14cc6a393a9453f8d495a, State: Initialized, Role: FOLLOWER
I20260929 19:03:08.451781 13655 consensus_queue.cc:260] T accba90b768d4e7e907312f7837ef0c4 P 961680b75ba14cc6a393a9453f8d495a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "961680b75ba14cc6a393a9453f8d495a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33043 } } peers { permanent_uuid: "e5dca27a04a448bb82a8bcaeeae3f252" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44183 } } peers { permanent_uuid: "6f7bc5348fe64b448f7da8c97840df38" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41345 } }
I20260929 19:03:08.451982 13655 ts_tablet_manager.cc:1428] T accba90b768d4e7e907312f7837ef0c4 P 961680b75ba14cc6a393a9453f8d495a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 19:03:08.452198 13620 heartbeater.cc:503] Master 127.0.98.62:37509 was elected leader, sending a full tablet report...
I20260929 19:03:08.472359 13663 raft_consensus.cc:482] T accba90b768d4e7e907312f7837ef0c4 P 961680b75ba14cc6a393a9453f8d495a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 19:03:08.472458 13663 raft_consensus.cc:504] T accba90b768d4e7e907312f7837ef0c4 P 961680b75ba14cc6a393a9453f8d495a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "961680b75ba14cc6a393a9453f8d495a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33043 } } peers { permanent_uuid: "e5dca27a04a448bb82a8bcaeeae3f252" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44183 } } peers { permanent_uuid: "6f7bc5348fe64b448f7da8c97840df38" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41345 } }
I20260929 19:03:08.472795 13663 leader_election.cc:288] T accba90b768d4e7e907312f7837ef0c4 P 961680b75ba14cc6a393a9453f8d495a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e5dca27a04a448bb82a8bcaeeae3f252 (127.0.98.1:44183), 6f7bc5348fe64b448f7da8c97840df38 (127.0.98.2:41345)
I20260929 19:03:08.485970 13651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:08.473497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46019 (local address 127.0.98.1:44183)
0929 19:03:08.473556 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:08.473557 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:08.477360 (+ 3803us) server_negotiation.cc:407] Connection header received
0929 19:03:08.477910 (+ 550us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:08.477911 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:08.477922 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:08.477947 (+ 25us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:03:08.479029 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:08.479685 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:08.484135 (+ 4450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:08.484721 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:08.484771 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:08.484782 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:08.485864 (+ 1082us) server_negotiation.cc:299] Negotiation successful
0929 19:03:08.485906 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260929 19:03:08.486428 13665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:08.473041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44183 (local address 127.0.98.3:46019)
0929 19:03:08.477289 (+ 4248us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:08.477301 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:08.477675 (+ 374us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:08.477998 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:08.477999 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:08.478000 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:03:08.478051 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:08.478052 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:08.480873 (+ 2821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:08.480874 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:08.483016 (+ 2142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:08.483019 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:08.484862 (+ 1843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:08.484862 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:08.484932 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:08.484936 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 19:03:08.486301 (+ 1365us) client_negotiation.cc:240] Negotiation successful
0929 19:03:08.486341 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4217,"thread_start_us":91,"threads_started":1}
I20260929 19:03:08.486735 13653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:08.473351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36475 (local address 127.0.98.2:41345)
0929 19:03:08.473398 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:08.473400 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:08.477460 (+ 4060us) server_negotiation.cc:407] Connection header received
0929 19:03:08.477585 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:08.477586 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:08.477601 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:08.477633 (+ 32us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:03:08.478117 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:08.478833 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:08.485022 (+ 6189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:08.485611 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:08.485654 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:08.485662 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:08.486635 (+ 973us) server_negotiation.cc:299] Negotiation successful
0929 19:03:08.486678 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260929 19:03:08.487288 13299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "accba90b768d4e7e907312f7837ef0c4" candidate_uuid: "961680b75ba14cc6a393a9453f8d495a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5dca27a04a448bb82a8bcaeeae3f252" is_pre_election: true
I20260929 19:03:08.487421 13299 raft_consensus.cc:2454] T accba90b768d4e7e907312f7837ef0c4 P e5dca27a04a448bb82a8bcaeeae3f252 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 961680b75ba14cc6a393a9453f8d495a in term 0.
I20260929 19:03:08.487650 13508 leader_election.cc:302] T accba90b768d4e7e907312f7837ef0c4 P 961680b75ba14cc6a393a9453f8d495a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 961680b75ba14cc6a393a9453f8d495a, e5dca27a04a448bb82a8bcaeeae3f252; no voters:
I20260929 19:03:08.487752 13663 raft_consensus.cc:2781] T accba90b768d4e7e907312f7837ef0c4 P 961680b75ba14cc6a393a9453f8d495a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 19:03:08.487803 13663 raft_consensus.cc:482] T accba90b768d4e7e907312f7837ef0c4 P 961680b75ba14cc6a393a9453f8d495a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 19:03:08.487823 13663 raft_consensus.cc:3037] T accba90b768d4e7e907312f7837ef0c4 P 961680b75ba14cc6a393a9453f8d495a [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:03:08.487026 13664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:08.473282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41345 (local address 127.0.98.3:36475)
0929 19:03:08.477406 (+ 4124us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:08.477416 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:08.477493 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:08.477755 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:08.477756 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:08.477757 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:03:08.477836 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:08.477838 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:08.479777 (+ 1939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:08.479778 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:08.484036 (+ 4258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:08.484039 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:08.485726 (+ 1687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:08.485727 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:08.485791 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:08.485794 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 19:03:08.486928 (+ 1134us) client_negotiation.cc:240] Negotiation successful
0929 19:03:08.486964 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4100,"thread_start_us":76,"threads_started":1}
I20260929 19:03:08.488680 13663 raft_consensus.cc:504] T accba90b768d4e7e907312f7837ef0c4 P 961680b75ba14cc6a393a9453f8d495a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "961680b75ba14cc6a393a9453f8d495a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33043 } } peers { permanent_uuid: "e5dca27a04a448bb82a8bcaeeae3f252" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44183 } } peers { permanent_uuid: "6f7bc5348fe64b448f7da8c97840df38" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41345 } }
I20260929 19:03:08.488795 13663 leader_election.cc:288] T accba90b768d4e7e907312f7837ef0c4 P 961680b75ba14cc6a393a9453f8d495a [CANDIDATE]: Term 1 election: Requested vote from peers e5dca27a04a448bb82a8bcaeeae3f252 (127.0.98.1:44183), 6f7bc5348fe64b448f7da8c97840df38 (127.0.98.2:41345)
I20260929 19:03:08.488956 13440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "accba90b768d4e7e907312f7837ef0c4" candidate_uuid: "961680b75ba14cc6a393a9453f8d495a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f7bc5348fe64b448f7da8c97840df38" is_pre_election: true
I20260929 19:03:08.489095 13440 raft_consensus.cc:2454] T accba90b768d4e7e907312f7837ef0c4 P 6f7bc5348fe64b448f7da8c97840df38 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 961680b75ba14cc6a393a9453f8d495a in term 0.
I20260929 19:03:08.489459 13299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "accba90b768d4e7e907312f7837ef0c4" candidate_uuid: "961680b75ba14cc6a393a9453f8d495a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5dca27a04a448bb82a8bcaeeae3f252"
I20260929 19:03:08.489523 13299 raft_consensus.cc:3037] T accba90b768d4e7e907312f7837ef0c4 P e5dca27a04a448bb82a8bcaeeae3f252 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:03:08.490288 13299 raft_consensus.cc:2454] T accba90b768d4e7e907312f7837ef0c4 P e5dca27a04a448bb82a8bcaeeae3f252 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 961680b75ba14cc6a393a9453f8d495a in term 1.
I20260929 19:03:08.490590 13508 leader_election.cc:302] T accba90b768d4e7e907312f7837ef0c4 P 961680b75ba14cc6a393a9453f8d495a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 961680b75ba14cc6a393a9453f8d495a, e5dca27a04a448bb82a8bcaeeae3f252; no voters:
I20260929 19:03:08.490717 13440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "accba90b768d4e7e907312f7837ef0c4" candidate_uuid: "961680b75ba14cc6a393a9453f8d495a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f7bc5348fe64b448f7da8c97840df38"
I20260929 19:03:08.490808 13440 raft_consensus.cc:3037] T accba90b768d4e7e907312f7837ef0c4 P 6f7bc5348fe64b448f7da8c97840df38 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:03:08.491559 13440 raft_consensus.cc:2454] T accba90b768d4e7e907312f7837ef0c4 P 6f7bc5348fe64b448f7da8c97840df38 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 961680b75ba14cc6a393a9453f8d495a in term 1.
I20260929 19:03:08.491739 13663 raft_consensus.cc:2781] T accba90b768d4e7e907312f7837ef0c4 P 961680b75ba14cc6a393a9453f8d495a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:03:08.491866 13663 raft_consensus.cc:686] T accba90b768d4e7e907312f7837ef0c4 P 961680b75ba14cc6a393a9453f8d495a [term 1 LEADER]: Becoming Leader. State: Replica: 961680b75ba14cc6a393a9453f8d495a, State: Running, Role: LEADER
I20260929 19:03:08.491941 13663 consensus_queue.cc:237] T accba90b768d4e7e907312f7837ef0c4 P 961680b75ba14cc6a393a9453f8d495a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "961680b75ba14cc6a393a9453f8d495a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33043 } } peers { permanent_uuid: "e5dca27a04a448bb82a8bcaeeae3f252" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44183 } } peers { permanent_uuid: "6f7bc5348fe64b448f7da8c97840df38" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41345 } }
I20260929 19:03:08.492652 13154 catalog_manager.cc:5495] T accba90b768d4e7e907312f7837ef0c4 P 961680b75ba14cc6a393a9453f8d495a reported cstate change: term changed from 0 to 1, leader changed from <none> to 961680b75ba14cc6a393a9453f8d495a (127.0.98.3). New cstate: current_term: 1 leader_uuid: "961680b75ba14cc6a393a9453f8d495a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "961680b75ba14cc6a393a9453f8d495a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33043 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e5dca27a04a448bb82a8bcaeeae3f252" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6f7bc5348fe64b448f7da8c97840df38" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41345 } health_report { overall_health: UNKNOWN } } }
I20260929 19:03:08.516669 13654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:08.513358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33568 (local address 127.0.98.3:33043)
0929 19:03:08.513417 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:08.513418 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:08.513459 (+ 41us) server_negotiation.cc:407] Connection header received
0929 19:03:08.513493 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:08.513494 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:08.513513 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:08.513590 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 19:03:08.513754 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:08.514492 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:08.515625 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:08.515924 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:08.515982 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:08.516158 (+ 176us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 19:03:08.516275 (+ 117us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 19:03:08.516337 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:08.516579 (+ 242us) server_negotiation.cc:299] Negotiation successful
0929 19:03:08.516599 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260929 19:03:08.518491 13440 raft_consensus.cc:1261] T accba90b768d4e7e907312f7837ef0c4 P 6f7bc5348fe64b448f7da8c97840df38 [term 1 FOLLOWER]: Refusing update from remote peer 961680b75ba14cc6a393a9453f8d495a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 19:03:08.518900 13666 consensus_queue.cc:1035] T accba90b768d4e7e907312f7837ef0c4 P 961680b75ba14cc6a393a9453f8d495a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f7bc5348fe64b448f7da8c97840df38" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 19:03:08.520956 13299 raft_consensus.cc:1261] T accba90b768d4e7e907312f7837ef0c4 P e5dca27a04a448bb82a8bcaeeae3f252 [term 1 FOLLOWER]: Refusing update from remote peer 961680b75ba14cc6a393a9453f8d495a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 19:03:08.521262 13666 consensus_queue.cc:1035] T accba90b768d4e7e907312f7837ef0c4 P 961680b75ba14cc6a393a9453f8d495a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5dca27a04a448bb82a8bcaeeae3f252" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 19:03:08.524731 13669 mvcc.cc:205] Tried to move back new op lower bound from 7334742378568798208 to 7334742378463346688. Current Snapshot: MvccSnapshot[applied={T|T < 7334742378568798208}]
I20260929 19:03:08.525264 13672 mvcc.cc:205] Tried to move back new op lower bound from 7334742378568798208 to 7334742378463346688. Current Snapshot: MvccSnapshot[applied={T|T < 7334742378568798208}]
I20260929 19:03:08.528785 13154 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40288:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 19:03:08.528890 13154 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40288:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 19:03:08.529160 13670 mvcc.cc:205] Tried to move back new op lower bound from 7334742378568798208 to 7334742378463346688. Current Snapshot: MvccSnapshot[applied={T|T < 7334742378568798208}]
I20260929 19:03:08.529634 13154 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5ed01c77975c4dc98943ebf46fb3f54f: Sending DeleteTablet for 3 replicas of tablet accba90b768d4e7e907312f7837ef0c4
I20260929 19:03:08.530040 13420 tablet_service.cc:1514] Processing DeleteTablet for tablet accba90b768d4e7e907312f7837ef0c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:03:08 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50514
I20260929 19:03:08.530244 13286 tablet_service.cc:1514] Processing DeleteTablet for tablet accba90b768d4e7e907312f7837ef0c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:03:08 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59950
I20260929 19:03:08.530563 13554 tablet_service.cc:1514] Processing DeleteTablet for tablet accba90b768d4e7e907312f7837ef0c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:03:08 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33552
I20260929 19:03:08.530794 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUbnLHE/build/release/bin/kudu with pid 13192
I20260929 19:03:08.537477 13678 tablet_replica.cc:330] T accba90b768d4e7e907312f7837ef0c4 P 6f7bc5348fe64b448f7da8c97840df38: stopping tablet replica
I20260929 19:03:08.537626 13678 raft_consensus.cc:2229] T accba90b768d4e7e907312f7837ef0c4 P 6f7bc5348fe64b448f7da8c97840df38 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 19:03:08.537786 13678 raft_consensus.cc:2258] T accba90b768d4e7e907312f7837ef0c4 P 6f7bc5348fe64b448f7da8c97840df38 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 19:03:08.538268 13678 ts_tablet_manager.cc:1905] T accba90b768d4e7e907312f7837ef0c4 P 6f7bc5348fe64b448f7da8c97840df38: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260929 19:03:08.542575 13139 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:44183 (error 108)
I20260929 19:03:08.542747 13680 tablet_replica.cc:330] T accba90b768d4e7e907312f7837ef0c4 P 961680b75ba14cc6a393a9453f8d495a: stopping tablet replica
I20260929 19:03:08.542847 13680 raft_consensus.cc:2229] T accba90b768d4e7e907312f7837ef0c4 P 961680b75ba14cc6a393a9453f8d495a [term 1 LEADER]: Raft consensus shutting down.
I20260929 19:03:08.542966 13680 raft_consensus.cc:2258] T accba90b768d4e7e907312f7837ef0c4 P 961680b75ba14cc6a393a9453f8d495a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 19:03:08.543287 13649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:08.543175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44183 (local address 127.0.0.1:59952)
0929 19:03:08.543217 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:08.543254 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
W20260929 19:03:08.543424 13139 catalog_manager.cc:4584] TS e5dca27a04a448bb82a8bcaeeae3f252 (127.0.98.1:44183): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet accba90b768d4e7e907312f7837ef0c4: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44183: connect: Connection refused (error 111)
I20260929 19:03:08.545295 13680 ts_tablet_manager.cc:1905] T accba90b768d4e7e907312f7837ef0c4 P 961680b75ba14cc6a393a9453f8d495a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 19:03:08.545382 13678 ts_tablet_manager.cc:1918] T accba90b768d4e7e907312f7837ef0c4 P 6f7bc5348fe64b448f7da8c97840df38: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 19:03:08.545478 13678 log.cc:1196] T accba90b768d4e7e907312f7837ef0c4 P 6f7bc5348fe64b448f7da8c97840df38: Deleting WAL directory at /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/minicluster-data/ts-1/wal/wals/accba90b768d4e7e907312f7837ef0c4
I20260929 19:03:08.545737 13678 ts_tablet_manager.cc:1939] T accba90b768d4e7e907312f7837ef0c4 P 6f7bc5348fe64b448f7da8c97840df38: Deleting consensus metadata
I20260929 19:03:08.547271 13680 ts_tablet_manager.cc:1918] T accba90b768d4e7e907312f7837ef0c4 P 961680b75ba14cc6a393a9453f8d495a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 19:03:08.548981 13680 log.cc:1196] T accba90b768d4e7e907312f7837ef0c4 P 961680b75ba14cc6a393a9453f8d495a: Deleting WAL directory at /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790708447806542-392-0/minicluster-data/ts-2/wal/wals/accba90b768d4e7e907312f7837ef0c4
I20260929 19:03:08.549279 13140 catalog_manager.cc:4841] TS 6f7bc5348fe64b448f7da8c97840df38 (127.0.98.2:41345): tablet accba90b768d4e7e907312f7837ef0c4 (table test-table [id=62cfa5512c4a4c7291a7ea703f4303db]) successfully deleted
I20260929 19:03:08.549311 13680 ts_tablet_manager.cc:1939] T accba90b768d4e7e907312f7837ef0c4 P 961680b75ba14cc6a393a9453f8d495a: Deleting consensus metadata
I20260929 19:03:08.549561 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUbnLHE/build/release/bin/kudu with pid 13356
I20260929 19:03:08.549652 13140 catalog_manager.cc:4841] TS 961680b75ba14cc6a393a9453f8d495a (127.0.98.3:33043): tablet accba90b768d4e7e907312f7837ef0c4 (table test-table [id=62cfa5512c4a4c7291a7ea703f4303db]) successfully deleted
I20260929 19:03:08.556452 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUbnLHE/build/release/bin/kudu with pid 13489
I20260929 19:03:08.561373 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUbnLHE/build/release/bin/kudu with pid 13121
I20260929 19:03:08.567054 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260929 19:03:08.902181 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260929 19:03:08.902256 392 mini_ranger.cc:66] Stopping Ranger...
I20260929 19:03:09.246578 392 mini_ranger.cc:68] Stopped Ranger
2026-09-29 19:03:09.246 UTC [12188] LOG: received smart shutdown request
2026-09-29 19:03:09.249 UTC [12188] LOG: background worker "logical replication launcher" (PID 12197) exited with exit code 1
2026-09-29 19:03:09.249 UTC [12192] LOG: shutting down
2026-09-29 19:03:09.400 UTC [12188] LOG: database system is shut down
2026-09-29T19:03:09Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (43350 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708447806542-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 19:03:09 dist-test-slave-lqkv krb5kdc[13690](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 19:03:09 dist-test-slave-lqkv krb5kdc[13690](info): set up 2 sockets
Sep 29 19:03:09 dist-test-slave-lqkv krb5kdc[13690](info): commencing operation
krb5kdc: starting...
W20260929 19:03:10.564569 392 mini_kdc.cc:121] Time spent starting KDC: real 1.044s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708447806542-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708447806542-392-0/krb5kdc/test-user.keytab.
Sep 29 19:03:10 dist-test-slave-lqkv krb5kdc[13690](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708590, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T19:03:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T19:03:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708447806542-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708447806542-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 19:03:10.665553 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUbnLHE/build/release/bin/kudu
/tmp/dist-test-taskUbnLHE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708447806542-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708447806542-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708447806542-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708447806542-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40931
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708447806542-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40931,127.0.98.61:43745,127.0.98.60:37331
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708447806542-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708447806542-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708447806542-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:03:10.742486 13703 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:03:10.742689 13703 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:03:10.742710 13703 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:03:10.743825 13703 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 19:03:10.743865 13703 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:03:10.743878 13703 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:03:10.743891 13703 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 19:03:10.743902 13703 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 19:03:10.745389 13703 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708447806542-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708447806542-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40931,127.0.98.61:43745,127.0.98.60:37331
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708447806542-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40931
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708447806542-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13703
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708447806542-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 18:09:13 UTC on ccycloud.quasar-hpkalf.root.comops.site
build id 4680913
I20260929 19:03:10.745590 13703 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:03:10.745754 13703 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:03:10.750460 13708 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:03:10.750504 13709 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:03:10.750429 13703 server_base.cc:1034] running on GCE node
W20260929 19:03:10.750438 13711 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:03:10.750989 13703 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:03:10.751235 13703 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:03:10.752372 13703 hybrid_clock.cc:648] HybridClock initialized: now 1790708590752363 us; error 26 us; skew 500 ppm
Sep 29 19:03:10 dist-test-slave-lqkv krb5kdc[13690](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708590, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:03:10.754760 13703 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 19:03:10.755173 13703 webserver.cc:460] Webserver started at http://127.0.98.62:46767/ using document root <none> and password file <none>
I20260929 19:03:10.755409 13703 fs_manager.cc:359] Metadata directory not provided
I20260929 19:03:10.755453 13703 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:03:10.755589 13703 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:03:10.756577 13703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708447806542-392-0/minicluster-data/master-0/data/instance:
uuid: "6356ae3cc45e4af6ab95dfd6c543852a"
format_stamp: "Formatted at 2026-09-29 19:03:10 on dist-test-slave-lqkv"
server_key: "f9185ce8226ddccab2585b1806109860"
server_key_iv: "5117ebd1a6a312bc14fc394083f086e8"
server_key_version: "encryptionkey@0"
I20260929 19:03:10.756953 13703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708447806542-392-0/minicluster-data/master-0/wal/instance:
uuid: "6356ae3cc45e4af6ab95dfd6c543852a"
format_stamp: "Formatted at 2026-09-29 19:03:10 on dist-test-slave-lqkv"
server_key: "f9185ce8226ddccab2585b1806109860"
server_key_iv: "5117ebd1a6a312bc14fc394083f086e8"
server_key_version: "encryptionkey@0"
I20260929 19:03:10.758237 13703 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260929 19:03:10.758899 13718 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:10.759083 13703 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 19:03:10.759150 13703 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708447806542-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708447806542-392-0/minicluster-data/master-0/wal
uuid: "6356ae3cc45e4af6ab95dfd6c543852a"
format_stamp: "Formatted at 2026-09-29 19:03:10 on dist-test-slave-lqkv"
server_key: "f9185ce8226ddccab2585b1806109860"
server_key_iv: "5117ebd1a6a312bc14fc394083f086e8"
server_key_version: "encryptionkey@0"
I20260929 19:03:10.759230 13703 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708447806542-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708447806542-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708447806542-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:03:10.780063 13703 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:03:10.782382 13703 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:03:10.782536 13703 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:03:10.786301 13703 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40931
I20260929 19:03:10.786346 13770 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40931 every 8 connection(s)
I20260929 19:03:10.786686 13703 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708447806542-392-0/minicluster-data/master-0/data/info.pb
I20260929 19:03:10.787461 13771 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:03:10.788977 13771 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40931 } has no permanent_uuid. Determining permanent_uuid...
I20260929 19:03:10.789386 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUbnLHE/build/release/bin/kudu as pid 13703
I20260929 19:03:10.789506 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708447806542-392-0/minicluster-data/master-0/wal/instance
I20260929 19:03:10.789732 392 external_mini_cluster.cc:1428] Setting key d33276c20847f6e0987271322c3ab24a
Sep 29 19:03:10 dist-test-slave-lqkv krb5kdc[13690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708590, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
I20260929 19:03:10.799448 13773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:10.789705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40931 (local address 127.0.0.1:41572)
0929 19:03:10.789896 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:10.789936 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:10.790430 (+ 494us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:10.790717 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:10.790719 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:10.791074 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:03:10.791297 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:10.791303 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:10.792385 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:10.792386 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:10.793584 (+ 1198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:10.793587 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:10.794076 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:10.794078 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:10.794179 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:10.794648 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:03:10.794657 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:03:10.796180 (+ 1523us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:03:10.797991 (+ 1811us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:03:10.797997 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:03:10.797999 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:03:10.798191 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:03:10.798464 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:03:10.798464 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:03:10.798465 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:03:10.798547 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:03:10.798893 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:03:10.798897 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:03:10.798953 (+ 56us) client_negotiation.cc:769] Sending connection context
0929 19:03:10.799063 (+ 110us) client_negotiation.cc:240] Negotiation successful
0929 19:03:10.799223 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":91,"threads_started":1}
I20260929 19:03:10.799448 13774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:10.789716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41572 (local address 127.0.98.62:40931)
0929 19:03:10.789904 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:10.789913 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:10.790045 (+ 132us) server_negotiation.cc:407] Connection header received
0929 19:03:10.790584 (+ 539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:10.790586 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:10.790604 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:10.790714 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:10.791414 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:10.792284 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:10.793673 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:10.793983 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:10.794072 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:10.796283 (+ 2211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:03:10.796290 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:03:10.796290 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:03:10.796297 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:03:10.797890 (+ 1593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:10.798311 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:10.798318 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:10.798318 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:10.798388 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:10.798630 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:10.798631 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:10.798631 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:10.798821 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:03:10.798913 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:10.799122 (+ 209us) server_negotiation.cc:299] Negotiation successful
0929 19:03:10.799280 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":39,"server-negotiator.queue_time_us":154,"thread_start_us":99,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
I20260929 19:03:10.800220 13771 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43745 } has no permanent_uuid. Determining permanent_uuid...
I20260929 19:03:10.800523 13773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:10.800427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43745 (local address 127.0.0.1:51268)
0929 19:03:10.800465 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:10.800503 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260929 19:03:10.800699 13719 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43745: connect: Connection refused (error 111)
I20260929 19:03:10.801162 13773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:10.801070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43745 (local address 127.0.0.1:51272)
0929 19:03:10.801119 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:10.801143 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260929 19:03:10.801266 13771 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:43745: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43745: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708447806542-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708447806542-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260929 19:03:10.813632 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUbnLHE/build/release/bin/kudu
/tmp/dist-test-taskUbnLHE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708447806542-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708447806542-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708447806542-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708447806542-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:43745
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708447806542-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40931,127.0.98.61:43745,127.0.98.60:37331
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708447806542-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708447806542-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708447806542-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 19:03:10.850493 13771 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43745 } attempt: 1
I20260929 19:03:10.851001 13773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:10.850866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43745 (local address 127.0.0.1:51280)
0929 19:03:10.850931 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:10.850972 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260929 19:03:10.851361 13773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:10.851279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43745 (local address 127.0.0.1:51290)
0929 19:03:10.851319 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:10.851343 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260929 19:03:10.851464 13771 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:43745: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43745: connect: Connection refused (error 111)
W20260929 19:03:10.891753 13778 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:03:10.891927 13778 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:03:10.891948 13778 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:03:10.893004 13778 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 19:03:10.893038 13778 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:03:10.893052 13778 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:03:10.893064 13778 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 19:03:10.893076 13778 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 19:03:10.894457 13778 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708447806542-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708447806542-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40931,127.0.98.61:43745,127.0.98.60:37331
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708447806542-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:43745
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708447806542-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13778
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708447806542-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 18:09:13 UTC on ccycloud.quasar-hpkalf.root.comops.site
build id 4680913
I20260929 19:03:10.894627 13778 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:03:10.894784 13778 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:03:10.899381 13786 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:03:10.899377 13783 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:03:10.899381 13784 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:03:10.899660 13778 server_base.cc:1034] running on GCE node
I20260929 19:03:10.899911 13778 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:03:10.900219 13778 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:03:10.901374 13778 hybrid_clock.cc:648] HybridClock initialized: now 1790708590901353 us; error 32 us; skew 500 ppm
Sep 29 19:03:10 dist-test-slave-lqkv krb5kdc[13690](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708590, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:03:10.903700 13778 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260929 19:03:10.904034 13778 webserver.cc:460] Webserver started at http://127.0.98.61:45755/ using document root <none> and password file <none>
I20260929 19:03:10.904243 13778 fs_manager.cc:359] Metadata directory not provided
I20260929 19:03:10.904284 13778 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:03:10.904366 13778 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:03:10.905337 13778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708447806542-392-0/minicluster-data/master-1/data/instance:
uuid: "7a1aaaddbc114b41b28522d5fcbde6e5"
format_stamp: "Formatted at 2026-09-29 19:03:10 on dist-test-slave-lqkv"
server_key: "31b87b4a648efabc3025d848b432bbbe"
server_key_iv: "72c7c9bc754a3c42acbe5399c9e2a601"
server_key_version: "encryptionkey@0"
I20260929 19:03:10.905611 13778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708447806542-392-0/minicluster-data/master-1/wal/instance:
uuid: "7a1aaaddbc114b41b28522d5fcbde6e5"
format_stamp: "Formatted at 2026-09-29 19:03:10 on dist-test-slave-lqkv"
server_key: "31b87b4a648efabc3025d848b432bbbe"
server_key_iv: "72c7c9bc754a3c42acbe5399c9e2a601"
server_key_version: "encryptionkey@0"
I20260929 19:03:10.906774 13778 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 19:03:10.907384 13793 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:10.907572 13778 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 19:03:10.907644 13778 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708447806542-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708447806542-392-0/minicluster-data/master-1/wal
uuid: "7a1aaaddbc114b41b28522d5fcbde6e5"
format_stamp: "Formatted at 2026-09-29 19:03:10 on dist-test-slave-lqkv"
server_key: "31b87b4a648efabc3025d848b432bbbe"
server_key_iv: "72c7c9bc754a3c42acbe5399c9e2a601"
server_key_version: "encryptionkey@0"
I20260929 19:03:10.907696 13778 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708447806542-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708447806542-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708447806542-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:03:10.919591 13771 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43745 } attempt: 2
I20260929 19:03:10.919987 13773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:10.919871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43745 (local address 127.0.0.1:51306)
0929 19:03:10.919927 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:10.919963 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260929 19:03:10.920351 13773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:10.920270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43745 (local address 127.0.0.1:51312)
0929 19:03:10.920311 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:10.920335 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260929 19:03:10.920455 13771 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:43745: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43745: connect: Connection refused (error 111)
I20260929 19:03:10.922384 13778 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:03:10.924010 13778 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:03:10.924187 13778 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:03:10.927786 13778 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:43745
I20260929 19:03:10.927817 13845 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:43745 every 8 connection(s)
I20260929 19:03:10.928121 13778 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708447806542-392-0/minicluster-data/master-1/data/info.pb
I20260929 19:03:10.928987 13846 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:03:10.930351 13846 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40931 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 19:03:10 dist-test-slave-lqkv krb5kdc[13690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708590, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 19:03:10.937639 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUbnLHE/build/release/bin/kudu as pid 13778
I20260929 19:03:10.937750 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708447806542-392-0/minicluster-data/master-1/wal/instance
I20260929 19:03:10.937942 392 external_mini_cluster.cc:1428] Setting key 1b9251604ea4d0961a0ff2629e189194
I20260929 19:03:10.938585 13774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:10.930955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41588 (local address 127.0.98.62:40931)
0929 19:03:10.931007 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:10.931010 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:10.931179 (+ 169us) server_negotiation.cc:407] Connection header received
0929 19:03:10.931643 (+ 464us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:10.931644 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:10.931658 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:10.931714 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:10.932304 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:10.932789 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:10.933790 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:10.934011 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:10.934087 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:10.936023 (+ 1936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:03:10.936024 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:03:10.936025 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:03:10.936032 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:03:10.937248 (+ 1216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:10.937630 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:10.937632 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:10.937632 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:10.937693 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:10.937969 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:10.937970 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:10.937971 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:10.938096 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:03:10.938169 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:10.938422 (+ 253us) server_negotiation.cc:299] Negotiation successful
0929 19:03:10.938506 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260929 19:03:10.938735 13848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:10.930917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40931 (local address 127.0.0.1:41588)
0929 19:03:10.931065 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:10.931105 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:10.931544 (+ 439us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:10.931774 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:10.931778 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:10.932038 (+ 260us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:03:10.932216 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:10.932220 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:10.932930 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:10.932931 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:10.933691 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:10.933692 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:10.934092 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:10.934094 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:10.934222 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:10.934690 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:03:10.934698 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:03:10.935922 (+ 1224us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:03:10.937329 (+ 1407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:03:10.937335 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:03:10.937337 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:03:10.937558 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:03:10.937759 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:03:10.937760 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:03:10.937761 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:03:10.937897 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:03:10.938161 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:03:10.938168 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:03:10.938295 (+ 127us) client_negotiation.cc:769] Sending connection context
0929 19:03:10.938389 (+ 94us) client_negotiation.cc:240] Negotiation successful
0929 19:03:10.938524 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":68,"threads_started":1}
I20260929 19:03:10.939240 13846 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43745 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 19:03:10 dist-test-slave-lqkv krb5kdc[13690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708590, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
I20260929 19:03:10.946827 13848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:10.939448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43745 (local address 127.0.0.1:51326)
0929 19:03:10.939496 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:10.939507 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:10.939555 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:10.939773 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:10.939774 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:10.939905 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:03:10.939942 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:10.939943 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:10.941179 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:10.941180 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:10.941654 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:10.941655 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:10.942298 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:10.942298 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:10.942351 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:10.942744 (+ 393us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:03:10.942747 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:03:10.943799 (+ 1052us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:03:10.945761 (+ 1962us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:03:10.945762 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:03:10.945762 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:03:10.945915 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:03:10.946144 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:03:10.946145 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:03:10.946145 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:03:10.946207 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:03:10.946470 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:03:10.946471 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:03:10.946506 (+ 35us) client_negotiation.cc:769] Sending connection context
0929 19:03:10.946592 (+ 86us) client_negotiation.cc:240] Negotiation successful
0929 19:03:10.946683 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260929 19:03:10.946915 13850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:10.939511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51326 (local address 127.0.98.61:43745)
0929 19:03:10.939645 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:10.939649 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:10.939672 (+ 23us) server_negotiation.cc:407] Connection header received
0929 19:03:10.939688 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:10.939689 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:10.939698 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:10.939773 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:10.940011 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:10.941102 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:10.941730 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:10.942210 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:10.942288 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:10.943885 (+ 1597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:03:10.943893 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:03:10.943893 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:03:10.943900 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:03:10.945682 (+ 1782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:10.946020 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:10.946023 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:10.946024 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:10.946077 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:10.946280 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:10.946281 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:10.946281 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:10.946401 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:03:10.946489 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:10.946622 (+ 133us) server_negotiation.cc:299] Negotiation successful
0929 19:03:10.946740 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":56,"server-negotiator.queue_time_us":111,"thread_start_us":71,"threads_started":1}
I20260929 19:03:10.947476 13846 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37331 } has no permanent_uuid. Determining permanent_uuid...
I20260929 19:03:10.947779 13848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:10.947664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37331 (local address 127.0.0.1:44670)
0929 19:03:10.947711 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:10.947750 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260929 19:03:10.947901 13795 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37331: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
I20260929 19:03:10.948303 13848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:10.948193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37331 (local address 127.0.0.1:44686)
0929 19:03:10.948239 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:10.948263 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260929 19:03:10.948392 13846 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:37331: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37331: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708447806542-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708447806542-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260929 19:03:10.961386 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUbnLHE/build/release/bin/kudu
/tmp/dist-test-taskUbnLHE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708447806542-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708447806542-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708447806542-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708447806542-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:37331
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708447806542-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40931,127.0.98.61:43745,127.0.98.60:37331
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708447806542-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708447806542-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708447806542-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 19:03:10.997617 13846 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37331 } attempt: 1
I20260929 19:03:10.998113 13848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:10.997972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37331 (local address 127.0.0.1:44692)
0929 19:03:10.998032 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:10.998078 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260929 19:03:10.998505 13848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:10.998394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37331 (local address 127.0.0.1:44708)
0929 19:03:10.998443 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:10.998468 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260929 19:03:10.998634 13846 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:37331: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37331: connect: Connection refused (error 111)
I20260929 19:03:11.011621 13771 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43745 } attempt: 3
Sep 29 19:03:11 dist-test-slave-lqkv krb5kdc[13690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708590, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260929 19:03:11.018668 13850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:11.011959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51332 (local address 127.0.98.61:43745)
0929 19:03:11.012015 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:11.012017 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:11.012027 (+ 10us) server_negotiation.cc:407] Connection header received
0929 19:03:11.012093 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:11.012094 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:11.012105 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:11.012164 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:11.012495 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:11.013123 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:11.013838 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:11.014058 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:11.014143 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:11.016152 (+ 2009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:03:11.016153 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:03:11.016154 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:03:11.016161 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:03:11.017574 (+ 1413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:11.017912 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:11.017914 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:11.017914 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:11.017968 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:11.018170 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:11.018171 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:11.018172 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:11.018284 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:03:11.018357 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:11.018505 (+ 148us) server_negotiation.cc:299] Negotiation successful
0929 19:03:11.018578 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260929 19:03:11.018671 13773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:11.011899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43745 (local address 127.0.0.1:51332)
0929 19:03:11.011960 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:11.011973 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:11.012025 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:11.012185 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:11.012186 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:11.012338 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:03:11.012396 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:11.012398 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:11.013214 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:11.013215 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:11.013750 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:11.013751 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:11.014140 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:11.014141 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:11.014223 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:11.014692 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:03:11.014696 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:03:11.016068 (+ 1372us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:03:11.017671 (+ 1603us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:03:11.017672 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:03:11.017672 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:03:11.017842 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:03:11.018035 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:03:11.018036 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:03:11.018037 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:03:11.018100 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:03:11.018351 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:03:11.018352 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:03:11.018398 (+ 46us) client_negotiation.cc:769] Sending connection context
0929 19:03:11.018482 (+ 84us) client_negotiation.cc:240] Negotiation successful
0929 19:03:11.018578 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260929 19:03:11.019352 13771 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37331 } has no permanent_uuid. Determining permanent_uuid...
I20260929 19:03:11.019732 13773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:11.019632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37331 (local address 127.0.0.1:44716)
0929 19:03:11.019684 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:11.019713 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260929 19:03:11.020048 13773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:11.019975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37331 (local address 127.0.0.1:44722)
0929 19:03:11.020011 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:11.020033 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260929 19:03:11.020169 13771 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:37331: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37331: connect: Connection refused (error 111)
W20260929 19:03:11.041831 13853 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:03:11.042001 13853 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:03:11.042021 13853 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:03:11.043040 13853 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 19:03:11.043071 13853 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:03:11.043084 13853 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:03:11.043097 13853 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 19:03:11.043115 13853 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 19:03:11.044480 13853 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708447806542-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708447806542-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40931,127.0.98.61:43745,127.0.98.60:37331
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708447806542-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:37331
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708447806542-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13853
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708447806542-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 18:09:13 UTC on ccycloud.quasar-hpkalf.root.comops.site
build id 4680913
I20260929 19:03:11.044646 13853 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:03:11.044816 13853 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:03:11.049398 13859 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:03:11.049557 13858 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:03:11.049600 13853 server_base.cc:1034] running on GCE node
W20260929 19:03:11.049564 13861 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:03:11.049994 13853 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:03:11.050238 13853 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:03:11.051297 13771 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37331 } attempt: 1
I20260929 19:03:11.051384 13853 hybrid_clock.cc:648] HybridClock initialized: now 1790708591051375 us; error 27 us; skew 500 ppm
I20260929 19:03:11.051633 13773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:11.051539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37331 (local address 127.0.0.1:44736)
0929 19:03:11.051586 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:11.051614 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260929 19:03:11.051914 13773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:11.051860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37331 (local address 127.0.0.1:44748)
0929 19:03:11.051881 (+ 21us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:11.051901 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":13}
W20260929 19:03:11.052008 13771 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:37331: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37331: connect: Connection refused (error 111)
Sep 29 19:03:11 dist-test-slave-lqkv krb5kdc[13690](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708591, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:03:11.053941 13853 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260929 19:03:11.054325 13853 webserver.cc:460] Webserver started at http://127.0.98.60:46055/ using document root <none> and password file <none>
I20260929 19:03:11.054533 13853 fs_manager.cc:359] Metadata directory not provided
I20260929 19:03:11.054595 13853 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:03:11.054739 13853 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:03:11.055727 13853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708447806542-392-0/minicluster-data/master-2/data/instance:
uuid: "f61783f16cef417e89386c0594d633ed"
format_stamp: "Formatted at 2026-09-29 19:03:11 on dist-test-slave-lqkv"
server_key: "890bb52c225825615e0bb84abe338ae8"
server_key_iv: "b522b381ac27b92a3eb3c317562c346f"
server_key_version: "encryptionkey@0"
I20260929 19:03:11.056068 13853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708447806542-392-0/minicluster-data/master-2/wal/instance:
uuid: "f61783f16cef417e89386c0594d633ed"
format_stamp: "Formatted at 2026-09-29 19:03:11 on dist-test-slave-lqkv"
server_key: "890bb52c225825615e0bb84abe338ae8"
server_key_iv: "b522b381ac27b92a3eb3c317562c346f"
server_key_version: "encryptionkey@0"
I20260929 19:03:11.057371 13853 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 19:03:11.058064 13868 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:11.058271 13853 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 19:03:11.058372 13853 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708447806542-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708447806542-392-0/minicluster-data/master-2/wal
uuid: "f61783f16cef417e89386c0594d633ed"
format_stamp: "Formatted at 2026-09-29 19:03:11 on dist-test-slave-lqkv"
server_key: "890bb52c225825615e0bb84abe338ae8"
server_key_iv: "b522b381ac27b92a3eb3c317562c346f"
server_key_version: "encryptionkey@0"
I20260929 19:03:11.058449 13853 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708447806542-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708447806542-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708447806542-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:03:11.066769 13846 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37331 } attempt: 2
I20260929 19:03:11.067083 13848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:11.066967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37331 (local address 127.0.0.1:44762)
0929 19:03:11.067019 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:11.067053 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260929 19:03:11.067413 13848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:11.067328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37331 (local address 127.0.0.1:44764)
0929 19:03:11.067370 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:11.067394 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260929 19:03:11.067512 13846 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:37331: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37331: connect: Connection refused (error 111)
I20260929 19:03:11.071314 13853 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:03:11.072966 13853 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:03:11.073150 13853 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:03:11.076625 13853 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:37331
I20260929 19:03:11.076653 13920 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:37331 every 8 connection(s)
I20260929 19:03:11.076989 13853 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708447806542-392-0/minicluster-data/master-2/data/info.pb
I20260929 19:03:11.077734 13921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:03:11.079056 13921 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40931 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 19:03:11 dist-test-slave-lqkv krb5kdc[13690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708591, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 19:03:11.085795 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUbnLHE/build/release/bin/kudu as pid 13853
I20260929 19:03:11.085929 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790708447806542-392-0/minicluster-data/master-2/wal/instance
I20260929 19:03:11.086107 392 external_mini_cluster.cc:1428] Setting key a3219f0608720f4b742192609419a0c2
I20260929 19:03:11.087616 13774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:11.079670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41590 (local address 127.0.98.62:40931)
0929 19:03:11.079725 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:11.079726 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:11.079896 (+ 170us) server_negotiation.cc:407] Connection header received
0929 19:03:11.080316 (+ 420us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:11.080317 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:11.080328 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:11.080378 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:11.080956 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:11.081410 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:11.082297 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:11.082482 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:11.082554 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:11.084494 (+ 1940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:03:11.084495 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:03:11.084496 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:03:11.084503 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:03:11.086044 (+ 1541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:11.086649 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:11.086650 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:11.086651 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:11.086716 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:11.086968 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:11.086969 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:11.086970 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:11.087113 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:03:11.087185 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:11.087468 (+ 283us) server_negotiation.cc:299] Negotiation successful
0929 19:03:11.087553 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260929 19:03:11.087769 13923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:11.079617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40931 (local address 127.0.0.1:41590)
0929 19:03:11.079799 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:11.079827 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:11.080219 (+ 392us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:11.080438 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:11.080441 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:11.080659 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:03:11.080881 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:11.080885 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:11.081493 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:11.081494 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:11.082203 (+ 709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:11.082205 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:11.082561 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:11.082562 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:11.082667 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:11.083063 (+ 396us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:03:11.083069 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:03:11.084400 (+ 1331us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:03:11.086133 (+ 1733us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:03:11.086139 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:03:11.086141 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:03:11.086370 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:03:11.086778 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:03:11.086778 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:03:11.086779 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:03:11.086903 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:03:11.087211 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:03:11.087215 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:03:11.087341 (+ 126us) client_negotiation.cc:769] Sending connection context
0929 19:03:11.087418 (+ 77us) client_negotiation.cc:240] Negotiation successful
0929 19:03:11.087558 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":87,"threads_started":1}
I20260929 19:03:11.088251 13921 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43745 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 19:03:11 dist-test-slave-lqkv krb5kdc[13690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708590, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 29 19:03:11 dist-test-slave-lqkv krb5kdc[13690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708591, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260929 19:03:11.094836 13923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:11.088474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43745 (local address 127.0.0.1:51342)
0929 19:03:11.088517 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:11.088526 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:11.088570 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:11.088690 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:11.088691 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:11.088873 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:03:11.088926 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:11.088927 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:11.089580 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:11.089581 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:11.090070 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:11.090071 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:11.090443 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:11.090444 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:11.090507 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:11.090880 (+ 373us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:03:11.090884 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:03:11.092490 (+ 1606us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:03:11.093855 (+ 1365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:03:11.093856 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:03:11.093856 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:03:11.094020 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:03:11.094218 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:03:11.094219 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:03:11.094219 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:03:11.094275 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:03:11.094523 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:03:11.094524 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:03:11.094564 (+ 40us) client_negotiation.cc:769] Sending connection context
0929 19:03:11.094657 (+ 93us) client_negotiation.cc:240] Negotiation successful
0929 19:03:11.094749 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260929 19:03:11.094846 13850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:11.088533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51342 (local address 127.0.98.61:43745)
0929 19:03:11.088579 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:11.088582 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:11.088602 (+ 20us) server_negotiation.cc:407] Connection header received
0929 19:03:11.088628 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:11.088629 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:11.088639 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:11.088680 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:11.089005 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:11.089506 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:11.090153 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:11.090361 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:11.090443 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:11.092573 (+ 2130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:03:11.092574 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:03:11.092575 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:03:11.092582 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:03:11.093773 (+ 1191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:11.094091 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:11.094092 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:11.094093 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:11.094151 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:11.094346 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:11.094346 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:11.094347 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:11.094452 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:03:11.094519 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:11.094684 (+ 165us) server_negotiation.cc:299] Negotiation successful
0929 19:03:11.094764 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260929 19:03:11.094980 13925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:11.086468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44768 (local address 127.0.98.60:37331)
0929 19:03:11.086606 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:11.086609 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:11.086661 (+ 52us) server_negotiation.cc:407] Connection header received
0929 19:03:11.086738 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:11.086740 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:11.086767 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:11.086840 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:11.087517 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:11.088549 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:11.089354 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:11.089643 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:11.089724 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:11.092133 (+ 2409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:03:11.092139 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:03:11.092139 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:03:11.092145 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:03:11.093785 (+ 1640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:11.094181 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:11.094184 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:11.094185 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:11.094244 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:11.094445 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:11.094446 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:11.094446 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:11.094573 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:03:11.094724 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:11.094800 (+ 76us) server_negotiation.cc:299] Negotiation successful
0929 19:03:11.094870 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":86,"threads_started":1}
I20260929 19:03:11.095568 13921 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37331 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 19:03:11 dist-test-slave-lqkv krb5kdc[13690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708591, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 29 19:03:11 dist-test-slave-lqkv krb5kdc[13690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708590, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 19:03:11.102142 13923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:11.095840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37331 (local address 127.0.0.1:44782)
0929 19:03:11.095936 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:11.095944 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:11.096005 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:11.096116 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:11.096117 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:11.096242 (+ 125us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:03:11.096263 (+ 21us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:11.096264 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:11.096903 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:11.096904 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:11.097353 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:11.097354 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:11.097751 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:11.097752 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:11.097799 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:11.098197 (+ 398us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:03:11.098201 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:03:11.099473 (+ 1272us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:03:11.101165 (+ 1692us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:03:11.101166 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:03:11.101167 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:03:11.101321 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:03:11.101514 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:03:11.101515 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:03:11.101515 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:03:11.101566 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:03:11.101799 (+ 233us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:03:11.101799 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:03:11.101834 (+ 35us) client_negotiation.cc:769] Sending connection context
0929 19:03:11.101900 (+ 66us) client_negotiation.cc:240] Negotiation successful
0929 19:03:11.102005 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":79}
I20260929 19:03:11.102154 13925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:11.095899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44782 (local address 127.0.98.60:37331)
0929 19:03:11.095931 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:11.095933 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:11.095993 (+ 60us) server_negotiation.cc:407] Connection header received
0929 19:03:11.096053 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:11.096054 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:11.096063 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:11.096105 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:11.096331 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:11.096784 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:11.097429 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:11.097664 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:11.097750 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:11.099559 (+ 1809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:03:11.099559 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:03:11.099560 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:03:11.099565 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:03:11.101068 (+ 1503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:11.101403 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:11.101404 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:11.101404 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:11.101454 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:11.101640 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:11.101641 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:11.101642 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:11.101738 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:03:11.101805 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:11.101928 (+ 123us) server_negotiation.cc:299] Negotiation successful
0929 19:03:11.102051 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":60,"server-negotiator.queue_time_us":17}
I20260929 19:03:11.103416 13774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:11.096033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41598 (local address 127.0.98.62:40931)
0929 19:03:11.096066 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:11.096068 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:11.096074 (+ 6us) server_negotiation.cc:407] Connection header received
0929 19:03:11.096092 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:11.096093 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:11.096100 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:11.096148 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:11.096654 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:11.097162 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:11.097803 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:11.098090 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:11.098160 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:11.100295 (+ 2135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:03:11.100296 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:03:11.100296 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:03:11.100303 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:03:11.102131 (+ 1828us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:11.102597 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:11.102598 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:11.102598 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:11.102655 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:11.102892 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:11.102893 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:11.102894 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:11.103025 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:03:11.103216 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:11.103254 (+ 38us) server_negotiation.cc:299] Negotiation successful
0929 19:03:11.103325 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260929 19:03:11.103849 13921 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f61783f16cef417e89386c0594d633ed: Bootstrap starting.
I20260929 19:03:11.104640 13921 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f61783f16cef417e89386c0594d633ed: Neither blocks nor log segments found. Creating new log.
I20260929 19:03:11.104983 13921 log.cc:824] T 00000000000000000000000000000000 P f61783f16cef417e89386c0594d633ed: Log is configured to *not* fsync() on all Append() calls
I20260929 19:03:11.105741 13921 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f61783f16cef417e89386c0594d633ed: No bootstrap required, opened a new log
I20260929 19:03:11.107849 13921 raft_consensus.cc:348] T 00000000000000000000000000000000 P f61783f16cef417e89386c0594d633ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6356ae3cc45e4af6ab95dfd6c543852a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40931 } } peers { permanent_uuid: "7a1aaaddbc114b41b28522d5fcbde6e5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43745 } } peers { permanent_uuid: "f61783f16cef417e89386c0594d633ed" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37331 } }
I20260929 19:03:11.107987 13921 raft_consensus.cc:374] T 00000000000000000000000000000000 P f61783f16cef417e89386c0594d633ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:03:11.108031 13921 raft_consensus.cc:729] T 00000000000000000000000000000000 P f61783f16cef417e89386c0594d633ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f61783f16cef417e89386c0594d633ed, State: Initialized, Role: FOLLOWER
I20260929 19:03:11.108140 13921 consensus_queue.cc:260] T 00000000000000000000000000000000 P f61783f16cef417e89386c0594d633ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6356ae3cc45e4af6ab95dfd6c543852a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40931 } } peers { permanent_uuid: "7a1aaaddbc114b41b28522d5fcbde6e5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43745 } } peers { permanent_uuid: "f61783f16cef417e89386c0594d633ed" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37331 } }
Sep 29 19:03:11 dist-test-slave-lqkv krb5kdc[13690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708590, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260929 19:03:11.108423 13927 sys_catalog.cc:455] T 00000000000000000000000000000000 P f61783f16cef417e89386c0594d633ed [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6356ae3cc45e4af6ab95dfd6c543852a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40931 } } peers { permanent_uuid: "7a1aaaddbc114b41b28522d5fcbde6e5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43745 } } peers { permanent_uuid: "f61783f16cef417e89386c0594d633ed" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37331 } } }
I20260929 19:03:11.108455 13921 sys_catalog.cc:564] T 00000000000000000000000000000000 P f61783f16cef417e89386c0594d633ed [sys.catalog]: configured and running, proceeding with master startup.
I20260929 19:03:11.108521 13927 sys_catalog.cc:458] T 00000000000000000000000000000000 P f61783f16cef417e89386c0594d633ed [sys.catalog]: This master's current role is: FOLLOWER
W20260929 19:03:11.110306 13938 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f61783f16cef417e89386c0594d633ed: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 19:03:11.110420 13938 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 19:03:11.110502 13853 master_runner.cc:186] Error getting master registration for 127.0.98.62:40931: OK, instance_id { permanent_uuid: "6356ae3cc45e4af6ab95dfd6c543852a" instance_seqno: 1790708590785261 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 19:03:11.112077 13853 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 19:03:11.112362 13850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:11.104027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51346 (local address 127.0.98.61:43745)
0929 19:03:11.104059 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:11.104061 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:11.104066 (+ 5us) server_negotiation.cc:407] Connection header received
0929 19:03:11.104126 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:11.104127 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:11.104134 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:11.104212 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:11.104658 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:11.105279 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:11.106146 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:11.106313 (+ 167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:11.106365 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:11.108725 (+ 2360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:03:11.108726 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:03:11.108726 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:03:11.108732 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:03:11.110449 (+ 1717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:11.110868 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:11.110869 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:11.110869 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:11.110948 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:11.111155 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:11.111156 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:11.111156 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:11.111271 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:03:11.111811 (+ 540us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:11.112221 (+ 410us) server_negotiation.cc:299] Negotiation successful
0929 19:03:11.112287 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260929 19:03:11.113559 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 19:03:11.122119 13774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:11.114836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41610 (local address 127.0.98.62:40931)
0929 19:03:11.114872 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:11.114874 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:11.114990 (+ 116us) server_negotiation.cc:407] Connection header received
0929 19:03:11.115106 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:11.115107 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:11.115118 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:11.115174 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:11.115851 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:11.116555 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:11.117591 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:11.117838 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:11.117876 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:11.119187 (+ 1311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:03:11.119188 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:03:11.119188 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:03:11.119194 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:03:11.120906 (+ 1712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:11.121348 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:11.121349 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:11.121350 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:11.121399 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:11.121591 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:11.121592 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:11.121593 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:11.121739 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:03:11.121853 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:11.121970 (+ 117us) server_negotiation.cc:299] Negotiation successful
0929 19:03:11.122038 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260929 19:03:11.122457 13925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:11.115025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44794 (local address 127.0.98.60:37331)
0929 19:03:11.115057 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:11.115059 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:11.115184 (+ 125us) server_negotiation.cc:407] Connection header received
0929 19:03:11.115304 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:11.115305 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:11.115315 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:11.115344 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:11.116205 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:11.116902 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:11.117926 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:11.118183 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:11.118229 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:11.119562 (+ 1333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:03:11.119564 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:03:11.119564 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:03:11.119570 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:03:11.121046 (+ 1476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:11.121715 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:11.121716 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:11.121716 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:11.121765 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:11.121958 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:11.121959 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:11.121959 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:11.122075 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:03:11.122152 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:11.122323 (+ 171us) server_negotiation.cc:299] Negotiation successful
0929 19:03:11.122385 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260929 19:03:11.123615 13850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:11.115129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51356 (local address 127.0.98.61:43745)
0929 19:03:11.115164 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:11.115166 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:11.115250 (+ 84us) server_negotiation.cc:407] Connection header received
0929 19:03:11.115338 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:11.115339 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:11.115348 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:11.115376 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:11.116198 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:11.116908 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:11.117926 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:11.118431 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:11.118470 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:11.120176 (+ 1706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:03:11.120177 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:03:11.120177 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:03:11.120183 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:03:11.122415 (+ 2232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:11.122947 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:11.122948 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:11.122948 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:11.123000 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:11.123173 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:11.123173 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:11.123174 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:11.123266 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:03:11.123374 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:11.123463 (+ 89us) server_negotiation.cc:299] Negotiation successful
0929 19:03:11.123528 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260929 19:03:11.127127 13771 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37331 } attempt: 2
Sep 29 19:03:11 dist-test-slave-lqkv krb5kdc[13690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708590, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260929 19:03:11.132997 13773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:11.127348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37331 (local address 127.0.0.1:44804)
0929 19:03:11.127393 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:11.127403 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:11.127439 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:11.127559 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:11.127559 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:11.127672 (+ 113us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:03:11.127699 (+ 27us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:11.127700 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:11.128444 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:11.128445 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:11.128912 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:11.128914 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:11.129351 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:11.129352 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:11.129403 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:11.129697 (+ 294us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:03:11.129701 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:03:11.130736 (+ 1035us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:03:11.132153 (+ 1417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:03:11.132154 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:03:11.132154 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:03:11.132299 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:03:11.132472 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:03:11.132473 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:03:11.132473 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:03:11.132516 (+ 43us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:03:11.132724 (+ 208us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:03:11.132725 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:03:11.132753 (+ 28us) client_negotiation.cc:769] Sending connection context
0929 19:03:11.132886 (+ 133us) client_negotiation.cc:240] Negotiation successful
0929 19:03:11.132944 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260929 19:03:11.133055 13925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:11.127418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44804 (local address 127.0.98.60:37331)
0929 19:03:11.127451 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:11.127453 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:11.127460 (+ 7us) server_negotiation.cc:407] Connection header received
0929 19:03:11.127494 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:11.127495 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:11.127503 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:11.127557 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:11.127757 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:11.128370 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:11.128975 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:11.129247 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:11.129307 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:11.130815 (+ 1508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:03:11.130816 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:03:11.130817 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:03:11.130823 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:03:11.132069 (+ 1246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:11.132361 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:11.132361 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:11.132362 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:11.132413 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:11.132566 (+ 153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:11.132567 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:11.132567 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:11.132671 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:03:11.132727 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:11.132912 (+ 185us) server_negotiation.cc:299] Negotiation successful
0929 19:03:11.132980 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260929 19:03:11.134298 13771 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6356ae3cc45e4af6ab95dfd6c543852a: Bootstrap starting.
I20260929 19:03:11.134964 13771 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6356ae3cc45e4af6ab95dfd6c543852a: Neither blocks nor log segments found. Creating new log.
I20260929 19:03:11.135288 13771 log.cc:824] T 00000000000000000000000000000000 P 6356ae3cc45e4af6ab95dfd6c543852a: Log is configured to *not* fsync() on all Append() calls
I20260929 19:03:11.136029 13771 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6356ae3cc45e4af6ab95dfd6c543852a: No bootstrap required, opened a new log
I20260929 19:03:11.137387 13771 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6356ae3cc45e4af6ab95dfd6c543852a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6356ae3cc45e4af6ab95dfd6c543852a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40931 } } peers { permanent_uuid: "7a1aaaddbc114b41b28522d5fcbde6e5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43745 } } peers { permanent_uuid: "f61783f16cef417e89386c0594d633ed" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37331 } }
I20260929 19:03:11.137507 13771 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6356ae3cc45e4af6ab95dfd6c543852a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:03:11.137552 13771 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6356ae3cc45e4af6ab95dfd6c543852a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6356ae3cc45e4af6ab95dfd6c543852a, State: Initialized, Role: FOLLOWER
I20260929 19:03:11.137670 13771 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6356ae3cc45e4af6ab95dfd6c543852a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6356ae3cc45e4af6ab95dfd6c543852a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40931 } } peers { permanent_uuid: "7a1aaaddbc114b41b28522d5fcbde6e5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43745 } } peers { permanent_uuid: "f61783f16cef417e89386c0594d633ed" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37331 } }
I20260929 19:03:11.137920 13948 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6356ae3cc45e4af6ab95dfd6c543852a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6356ae3cc45e4af6ab95dfd6c543852a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40931 } } peers { permanent_uuid: "7a1aaaddbc114b41b28522d5fcbde6e5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43745 } } peers { permanent_uuid: "f61783f16cef417e89386c0594d633ed" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37331 } } }
I20260929 19:03:11.137962 13771 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6356ae3cc45e4af6ab95dfd6c543852a [sys.catalog]: configured and running, proceeding with master startup.
I20260929 19:03:11.138010 13948 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6356ae3cc45e4af6ab95dfd6c543852a [sys.catalog]: This master's current role is: FOLLOWER
W20260929 19:03:11.139616 13959 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6356ae3cc45e4af6ab95dfd6c543852a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 19:03:11.139917 13959 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 19:03:11.140453 13703 master_runner.cc:186] Error getting master registration for 127.0.98.61:43745: OK, instance_id { permanent_uuid: "7a1aaaddbc114b41b28522d5fcbde6e5" instance_seqno: 1790708590926787 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 19:03:11.142298 13703 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 19:03:11.158718 13846 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37331 } attempt: 3
Sep 29 19:03:11 dist-test-slave-lqkv krb5kdc[13690](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708590, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260929 19:03:11.164610 13925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:11.158985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44820 (local address 127.0.98.60:37331)
0929 19:03:11.159018 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:11.159020 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:11.159028 (+ 8us) server_negotiation.cc:407] Connection header received
0929 19:03:11.159099 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:11.159100 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:11.159114 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:11.159171 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:11.159414 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:11.159811 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:11.160402 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:11.160563 (+ 161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:11.160629 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:11.162331 (+ 1702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:03:11.162332 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:03:11.162333 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:03:11.162338 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:03:11.163577 (+ 1239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:11.163891 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:11.163892 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:11.163892 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:11.163945 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:11.164147 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:11.164147 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:11.164148 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:11.164246 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:03:11.164315 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:11.164463 (+ 148us) server_negotiation.cc:299] Negotiation successful
0929 19:03:11.164530 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260929 19:03:11.164614 13848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:11.158955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37331 (local address 127.0.0.1:44820)
0929 19:03:11.158996 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:11.159006 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:11.159042 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:11.159174 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:11.159175 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:11.159313 (+ 138us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:03:11.159345 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:11.159346 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:11.159883 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:11.159884 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:11.160327 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:11.160328 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:11.160627 (+ 299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:11.160628 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:11.160679 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:11.161134 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:03:11.161139 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:03:11.162260 (+ 1121us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:03:11.163672 (+ 1412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:03:11.163673 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:03:11.163673 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:03:11.163824 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:03:11.164005 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:03:11.164006 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:03:11.164006 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:03:11.164083 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:03:11.164321 (+ 238us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:03:11.164322 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:03:11.164360 (+ 38us) client_negotiation.cc:769] Sending connection context
0929 19:03:11.164442 (+ 82us) client_negotiation.cc:240] Negotiation successful
0929 19:03:11.164530 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260929 19:03:11.165915 13846 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a1aaaddbc114b41b28522d5fcbde6e5: Bootstrap starting.
I20260929 19:03:11.166461 13846 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7a1aaaddbc114b41b28522d5fcbde6e5: Neither blocks nor log segments found. Creating new log.
I20260929 19:03:11.166755 13846 log.cc:824] T 00000000000000000000000000000000 P 7a1aaaddbc114b41b28522d5fcbde6e5: Log is configured to *not* fsync() on all Append() calls
I20260929 19:03:11.167687 13846 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a1aaaddbc114b41b28522d5fcbde6e5: No bootstrap required, opened a new log
I20260929 19:03:11.168972 13846 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7a1aaaddbc114b41b28522d5fcbde6e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6356ae3cc45e4af6ab95dfd6c543852a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40931 } } peers { permanent_uuid: "7a1aaaddbc114b41b28522d5fcbde6e5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43745 } } peers { permanent_uuid: "f61783f16cef417e89386c0594d633ed" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37331 } }
I20260929 19:03:11.169081 13846 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7a1aaaddbc114b41b28522d5fcbde6e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:03:11.169147 13846 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7a1aaaddbc114b41b28522d5fcbde6e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a1aaaddbc114b41b28522d5fcbde6e5, State: Initialized, Role: FOLLOWER
I20260929 19:03:11.169284 13846 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7a1aaaddbc114b41b28522d5fcbde6e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6356ae3cc45e4af6ab95dfd6c543852a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40931 } } peers { permanent_uuid: "7a1aaaddbc114b41b28522d5fcbde6e5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43745 } } peers { permanent_uuid: "f61783f16cef417e89386c0594d633ed" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37331 } }
I20260929 19:03:11.169512 13962 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a1aaaddbc114b41b28522d5fcbde6e5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6356ae3cc45e4af6ab95dfd6c543852a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40931 } } peers { permanent_uuid: "7a1aaaddbc114b41b28522d5fcbde6e5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43745 } } peers { permanent_uuid: "f61783f16cef417e89386c0594d633ed" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37331 } } }
I20260929 19:03:11.169561 13846 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7a1aaaddbc114b41b28522d5fcbde6e5 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 19:03:11.169627 13962 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a1aaaddbc114b41b28522d5fcbde6e5 [sys.catalog]: This master's current role is: FOLLOWER
W20260929 19:03:11.171334 13973 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7a1aaaddbc114b41b28522d5fcbde6e5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 19:03:11.171422 13973 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 19:03:11.173311 13778 master_runner.cc:263] No leader master found from master 7a1aaaddbc114b41b28522d5fcbde6e5
I20260929 19:03:11.173352 13778 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260929 19:03:11.174757 13799 tablet.cc:2369] T 00000000000000000000000000000000 P 7a1aaaddbc114b41b28522d5fcbde6e5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 19:03:11.243173 13948 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6356ae3cc45e4af6ab95dfd6c543852a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 19:03:11.243301 13948 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6356ae3cc45e4af6ab95dfd6c543852a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6356ae3cc45e4af6ab95dfd6c543852a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40931 } } peers { permanent_uuid: "7a1aaaddbc114b41b28522d5fcbde6e5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43745 } } peers { permanent_uuid: "f61783f16cef417e89386c0594d633ed" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37331 } }
I20260929 19:03:11.243498 13948 leader_election.cc:288] T 00000000000000000000000000000000 P 6356ae3cc45e4af6ab95dfd6c543852a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7a1aaaddbc114b41b28522d5fcbde6e5 (127.0.98.61:43745), f61783f16cef417e89386c0594d633ed (127.0.98.60:37331)
I20260929 19:03:11.243980 13820 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6356ae3cc45e4af6ab95dfd6c543852a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a1aaaddbc114b41b28522d5fcbde6e5" is_pre_election: true
I20260929 19:03:11.244144 13820 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7a1aaaddbc114b41b28522d5fcbde6e5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6356ae3cc45e4af6ab95dfd6c543852a in term 0.
I20260929 19:03:11.244458 13719 leader_election.cc:302] T 00000000000000000000000000000000 P 6356ae3cc45e4af6ab95dfd6c543852a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6356ae3cc45e4af6ab95dfd6c543852a, 7a1aaaddbc114b41b28522d5fcbde6e5; no voters:
I20260929 19:03:11.244831 13895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6356ae3cc45e4af6ab95dfd6c543852a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f61783f16cef417e89386c0594d633ed" is_pre_election: true
I20260929 19:03:11.244968 13895 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f61783f16cef417e89386c0594d633ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6356ae3cc45e4af6ab95dfd6c543852a in term 0.
I20260929 19:03:11.245252 13948 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6356ae3cc45e4af6ab95dfd6c543852a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 19:03:11.245306 13948 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6356ae3cc45e4af6ab95dfd6c543852a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 19:03:11.245329 13948 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6356ae3cc45e4af6ab95dfd6c543852a [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:03:11.246250 13948 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6356ae3cc45e4af6ab95dfd6c543852a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6356ae3cc45e4af6ab95dfd6c543852a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40931 } } peers { permanent_uuid: "7a1aaaddbc114b41b28522d5fcbde6e5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43745 } } peers { permanent_uuid: "f61783f16cef417e89386c0594d633ed" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37331 } }
I20260929 19:03:11.246361 13948 leader_election.cc:288] T 00000000000000000000000000000000 P 6356ae3cc45e4af6ab95dfd6c543852a [CANDIDATE]: Term 1 election: Requested vote from peers 7a1aaaddbc114b41b28522d5fcbde6e5 (127.0.98.61:43745), f61783f16cef417e89386c0594d633ed (127.0.98.60:37331)
I20260929 19:03:11.246635 13820 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6356ae3cc45e4af6ab95dfd6c543852a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a1aaaddbc114b41b28522d5fcbde6e5"
I20260929 19:03:11.246712 13820 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7a1aaaddbc114b41b28522d5fcbde6e5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:03:11.247504 13820 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7a1aaaddbc114b41b28522d5fcbde6e5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6356ae3cc45e4af6ab95dfd6c543852a in term 1.
I20260929 19:03:11.247677 13719 leader_election.cc:302] T 00000000000000000000000000000000 P 6356ae3cc45e4af6ab95dfd6c543852a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6356ae3cc45e4af6ab95dfd6c543852a, 7a1aaaddbc114b41b28522d5fcbde6e5; no voters:
I20260929 19:03:11.247880 13895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6356ae3cc45e4af6ab95dfd6c543852a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f61783f16cef417e89386c0594d633ed"
I20260929 19:03:11.247956 13895 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f61783f16cef417e89386c0594d633ed [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:03:11.248685 13895 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f61783f16cef417e89386c0594d633ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6356ae3cc45e4af6ab95dfd6c543852a in term 1.
I20260929 19:03:11.248941 13948 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6356ae3cc45e4af6ab95dfd6c543852a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:03:11.249100 13948 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6356ae3cc45e4af6ab95dfd6c543852a [term 1 LEADER]: Becoming Leader. State: Replica: 6356ae3cc45e4af6ab95dfd6c543852a, State: Running, Role: LEADER
I20260929 19:03:11.249177 13948 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6356ae3cc45e4af6ab95dfd6c543852a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6356ae3cc45e4af6ab95dfd6c543852a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40931 } } peers { permanent_uuid: "7a1aaaddbc114b41b28522d5fcbde6e5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43745 } } peers { permanent_uuid: "f61783f16cef417e89386c0594d633ed" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37331 } }
I20260929 19:03:11.249490 13948 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6356ae3cc45e4af6ab95dfd6c543852a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6356ae3cc45e4af6ab95dfd6c543852a. Latest consensus state: current_term: 1 leader_uuid: "6356ae3cc45e4af6ab95dfd6c543852a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6356ae3cc45e4af6ab95dfd6c543852a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40931 } } peers { permanent_uuid: "7a1aaaddbc114b41b28522d5fcbde6e5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43745 } } peers { permanent_uuid: "f61783f16cef417e89386c0594d633ed" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37331 } } }
I20260929 19:03:11.249549 13948 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6356ae3cc45e4af6ab95dfd6c543852a [sys.catalog]: This master's current role is: LEADER
I20260929 19:03:11.249832 13977 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 19:03:11.250030 13977 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 19:03:11.251855 13895 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f61783f16cef417e89386c0594d633ed [term 1 FOLLOWER]: Refusing update from remote peer 6356ae3cc45e4af6ab95dfd6c543852a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 19:03:11.252053 13820 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7a1aaaddbc114b41b28522d5fcbde6e5 [term 1 FOLLOWER]: Refusing update from remote peer 6356ae3cc45e4af6ab95dfd6c543852a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 19:03:11.252519 13948 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6356ae3cc45e4af6ab95dfd6c543852a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f61783f16cef417e89386c0594d633ed" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 19:03:11.252648 13948 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6356ae3cc45e4af6ab95dfd6c543852a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a1aaaddbc114b41b28522d5fcbde6e5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 19:03:11.253943 13962 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a1aaaddbc114b41b28522d5fcbde6e5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6356ae3cc45e4af6ab95dfd6c543852a. Latest consensus state: current_term: 1 leader_uuid: "6356ae3cc45e4af6ab95dfd6c543852a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6356ae3cc45e4af6ab95dfd6c543852a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40931 } } peers { permanent_uuid: "7a1aaaddbc114b41b28522d5fcbde6e5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43745 } } peers { permanent_uuid: "f61783f16cef417e89386c0594d633ed" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37331 } } }
I20260929 19:03:11.254047 13962 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a1aaaddbc114b41b28522d5fcbde6e5 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 19:03:11.255940 13927 sys_catalog.cc:455] T 00000000000000000000000000000000 P f61783f16cef417e89386c0594d633ed [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6356ae3cc45e4af6ab95dfd6c543852a. Latest consensus state: current_term: 1 leader_uuid: "6356ae3cc45e4af6ab95dfd6c543852a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6356ae3cc45e4af6ab95dfd6c543852a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40931 } } peers { permanent_uuid: "7a1aaaddbc114b41b28522d5fcbde6e5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43745 } } peers { permanent_uuid: "f61783f16cef417e89386c0594d633ed" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37331 } } }
I20260929 19:03:11.256018 13927 sys_catalog.cc:458] T 00000000000000000000000000000000 P f61783f16cef417e89386c0594d633ed [sys.catalog]: This master's current role is: FOLLOWER
I20260929 19:03:11.257493 13948 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6356ae3cc45e4af6ab95dfd6c543852a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6356ae3cc45e4af6ab95dfd6c543852a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6356ae3cc45e4af6ab95dfd6c543852a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40931 } } peers { permanent_uuid: "7a1aaaddbc114b41b28522d5fcbde6e5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43745 } } peers { permanent_uuid: "f61783f16cef417e89386c0594d633ed" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37331 } } }
I20260929 19:03:11.257563 13948 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6356ae3cc45e4af6ab95dfd6c543852a [sys.catalog]: This master's current role is: LEADER
I20260929 19:03:11.257993 13962 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a1aaaddbc114b41b28522d5fcbde6e5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6356ae3cc45e4af6ab95dfd6c543852a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6356ae3cc45e4af6ab95dfd6c543852a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40931 } } peers { permanent_uuid: "7a1aaaddbc114b41b28522d5fcbde6e5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43745 } } peers { permanent_uuid: "f61783f16cef417e89386c0594d633ed" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37331 } } }
I20260929 19:03:11.258065 13962 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a1aaaddbc114b41b28522d5fcbde6e5 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 19:03:11.258299 13978 mvcc.cc:205] Tried to move back new op lower bound from 7334742389765042176 to 7334742389756878848. Current Snapshot: MvccSnapshot[applied={T|T < 7334742389765042176}]
I20260929 19:03:11.258834 13979 mvcc.cc:205] Tried to move back new op lower bound from 7334742389765042176 to 7334742389756878848. Current Snapshot: MvccSnapshot[applied={T|T < 7334742389765042176}]
I20260929 19:03:11.259270 13948 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6356ae3cc45e4af6ab95dfd6c543852a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6356ae3cc45e4af6ab95dfd6c543852a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6356ae3cc45e4af6ab95dfd6c543852a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40931 } } peers { permanent_uuid: "7a1aaaddbc114b41b28522d5fcbde6e5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43745 } } peers { permanent_uuid: "f61783f16cef417e89386c0594d633ed" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37331 } } }
I20260929 19:03:11.259341 13948 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6356ae3cc45e4af6ab95dfd6c543852a [sys.catalog]: This master's current role is: LEADER
I20260929 19:03:11.259732 13927 sys_catalog.cc:455] T 00000000000000000000000000000000 P f61783f16cef417e89386c0594d633ed [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6356ae3cc45e4af6ab95dfd6c543852a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6356ae3cc45e4af6ab95dfd6c543852a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40931 } } peers { permanent_uuid: "7a1aaaddbc114b41b28522d5fcbde6e5" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43745 } } peers { permanent_uuid: "f61783f16cef417e89386c0594d633ed" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37331 } } }
I20260929 19:03:11.259800 13927 sys_catalog.cc:458] T 00000000000000000000000000000000 P f61783f16cef417e89386c0594d633ed [sys.catalog]: This master's current role is: FOLLOWER
I20260929 19:03:11.260409 13981 mvcc.cc:205] Tried to move back new op lower bound from 7334742389765042176 to 7334742389756878848. Current Snapshot: MvccSnapshot[applied={T|T < 7334742389765042176}]
I20260929 19:03:11.261380 13977 catalog_manager.cc:1347] Generated new cluster ID: 4faf9fadd09145b9b7dba60cf193f88c
I20260929 19:03:11.261420 13977 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 19:03:11.289827 13977 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 19:03:11.290503 13977 catalog_manager.cc:1504] Loading token signing keys...
I20260929 19:03:11.299629 13977 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6356ae3cc45e4af6ab95dfd6c543852a: Generated new TSK 0
I20260929 19:03:11.299867 13977 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 19:03:12.110816 13938 catalog_manager.cc:1259] Loaded cluster ID: 4faf9fadd09145b9b7dba60cf193f88c
I20260929 19:03:12.110888 13938 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f61783f16cef417e89386c0594d633ed: loading cluster ID for follower catalog manager: success
I20260929 19:03:12.111634 13938 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f61783f16cef417e89386c0594d633ed: acquiring CA information for follower catalog manager: success
I20260929 19:03:12.111902 13938 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f61783f16cef417e89386c0594d633ed: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 19:03:12.171826 13973 catalog_manager.cc:1259] Loaded cluster ID: 4faf9fadd09145b9b7dba60cf193f88c
I20260929 19:03:12.171880 13973 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7a1aaaddbc114b41b28522d5fcbde6e5: loading cluster ID for follower catalog manager: success
I20260929 19:03:12.172684 13973 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7a1aaaddbc114b41b28522d5fcbde6e5: acquiring CA information for follower catalog manager: success
I20260929 19:03:12.173003 13973 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7a1aaaddbc114b41b28522d5fcbde6e5: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 19:03:12.347891 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUbnLHE/build/release/bin/kudu with pid 13703
I20260929 19:03:12.352982 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUbnLHE/build/release/bin/kudu with pid 13778
I20260929 19:03:12.357625 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUbnLHE/build/release/bin/kudu with pid 13853
2026-09-29T19:03:12Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (2855 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790708447806542-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 19:03:12 dist-test-slave-lqkv krb5kdc[13993](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 19:03:12 dist-test-slave-lqkv krb5kdc[13993](info): set up 2 sockets
Sep 29 19:03:12 dist-test-slave-lqkv krb5kdc[13993](info): commencing operation
krb5kdc: starting...
W20260929 19:03:13.352474 392 mini_kdc.cc:121] Time spent starting KDC: real 0.976s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790708447806542-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790708447806542-392-0/krb5kdc/test-user.keytab.
Sep 29 19:03:13 dist-test-slave-lqkv krb5kdc[13993](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708593, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T19:03:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T19:03:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790708447806542-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790708447806542-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 19:03:13.458847 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUbnLHE/build/release/bin/kudu
/tmp/dist-test-taskUbnLHE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790708447806542-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790708447806542-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790708447806542-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790708447806542-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34727
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790708447806542-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34727
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790708447806542-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790708447806542-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790708447806542-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:03:13.532199 14006 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:03:13.532410 14006 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:03:13.532444 14006 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:03:13.533566 14006 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 19:03:13.533629 14006 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:03:13.533651 14006 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:03:13.533670 14006 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 19:03:13.533720 14006 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 19:03:13.535125 14006 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790708447806542-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790708447806542-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34727
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790708447806542-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34727
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790708447806542-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14006
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790708447806542-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 18:09:13 UTC on ccycloud.quasar-hpkalf.root.comops.site
build id 4680913
I20260929 19:03:13.535374 14006 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:03:13.535583 14006 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:03:13.540187 14014 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:03:13.540269 14006 server_base.cc:1034] running on GCE node
W20260929 19:03:13.540166 14011 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:03:13.540170 14012 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:03:13.540620 14006 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:03:13.540949 14006 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:03:13.542109 14006 hybrid_clock.cc:648] HybridClock initialized: now 1790708593542080 us; error 41 us; skew 500 ppm
Sep 29 19:03:13 dist-test-slave-lqkv krb5kdc[13993](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708593, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:03:13.544381 14006 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 19:03:13.544778 14006 webserver.cc:460] Webserver started at http://127.0.98.62:43471/ using document root <none> and password file <none>
I20260929 19:03:13.545020 14006 fs_manager.cc:359] Metadata directory not provided
I20260929 19:03:13.545063 14006 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:03:13.545148 14006 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:03:13.546092 14006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790708447806542-392-0/minicluster-data/master-0/data/instance:
uuid: "e72376e27ef742698d4b6de6ddf850fa"
format_stamp: "Formatted at 2026-09-29 19:03:13 on dist-test-slave-lqkv"
server_key: "c56116cfd7b2e6dba70f569596bc876e"
server_key_iv: "45e65bb1a65e31f570a043a70ea0b8cb"
server_key_version: "encryptionkey@0"
I20260929 19:03:13.546375 14006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790708447806542-392-0/minicluster-data/master-0/wal/instance:
uuid: "e72376e27ef742698d4b6de6ddf850fa"
format_stamp: "Formatted at 2026-09-29 19:03:13 on dist-test-slave-lqkv"
server_key: "c56116cfd7b2e6dba70f569596bc876e"
server_key_iv: "45e65bb1a65e31f570a043a70ea0b8cb"
server_key_version: "encryptionkey@0"
I20260929 19:03:13.547545 14006 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 19:03:13.548144 14021 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:13.548296 14006 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 19:03:13.548390 14006 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790708447806542-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790708447806542-392-0/minicluster-data/master-0/wal
uuid: "e72376e27ef742698d4b6de6ddf850fa"
format_stamp: "Formatted at 2026-09-29 19:03:13 on dist-test-slave-lqkv"
server_key: "c56116cfd7b2e6dba70f569596bc876e"
server_key_iv: "45e65bb1a65e31f570a043a70ea0b8cb"
server_key_version: "encryptionkey@0"
I20260929 19:03:13.548465 14006 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790708447806542-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790708447806542-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790708447806542-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:03:13.560173 14006 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:03:13.561864 14006 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:03:13.562016 14006 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:03:13.565761 14006 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34727
I20260929 19:03:13.565810 14073 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34727 every 8 connection(s)
I20260929 19:03:13.566139 14006 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790708447806542-392-0/minicluster-data/master-0/data/info.pb
I20260929 19:03:13.566915 14074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:03:13.569020 14074 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e72376e27ef742698d4b6de6ddf850fa: Bootstrap starting.
I20260929 19:03:13.569547 14074 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e72376e27ef742698d4b6de6ddf850fa: Neither blocks nor log segments found. Creating new log.
I20260929 19:03:13.569818 14074 log.cc:824] T 00000000000000000000000000000000 P e72376e27ef742698d4b6de6ddf850fa: Log is configured to *not* fsync() on all Append() calls
I20260929 19:03:13.570432 14074 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e72376e27ef742698d4b6de6ddf850fa: No bootstrap required, opened a new log
I20260929 19:03:13.571870 14074 raft_consensus.cc:348] T 00000000000000000000000000000000 P e72376e27ef742698d4b6de6ddf850fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e72376e27ef742698d4b6de6ddf850fa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34727 } }
I20260929 19:03:13.571992 14074 raft_consensus.cc:374] T 00000000000000000000000000000000 P e72376e27ef742698d4b6de6ddf850fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:03:13.572037 14074 raft_consensus.cc:729] T 00000000000000000000000000000000 P e72376e27ef742698d4b6de6ddf850fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72376e27ef742698d4b6de6ddf850fa, State: Initialized, Role: FOLLOWER
I20260929 19:03:13.572173 14074 consensus_queue.cc:260] T 00000000000000000000000000000000 P e72376e27ef742698d4b6de6ddf850fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e72376e27ef742698d4b6de6ddf850fa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34727 } }
I20260929 19:03:13.572232 14074 raft_consensus.cc:388] T 00000000000000000000000000000000 P e72376e27ef742698d4b6de6ddf850fa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 19:03:13.572252 14074 raft_consensus.cc:482] T 00000000000000000000000000000000 P e72376e27ef742698d4b6de6ddf850fa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 19:03:13.572321 14074 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e72376e27ef742698d4b6de6ddf850fa [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:03:13.572803 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUbnLHE/build/release/bin/kudu as pid 14006
I20260929 19:03:13.572902 14074 raft_consensus.cc:504] T 00000000000000000000000000000000 P e72376e27ef742698d4b6de6ddf850fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e72376e27ef742698d4b6de6ddf850fa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34727 } }
I20260929 19:03:13.572968 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790708447806542-392-0/minicluster-data/master-0/wal/instance
I20260929 19:03:13.573021 14074 leader_election.cc:302] T 00000000000000000000000000000000 P e72376e27ef742698d4b6de6ddf850fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e72376e27ef742698d4b6de6ddf850fa; no voters:
I20260929 19:03:13.573184 392 external_mini_cluster.cc:1428] Setting key ef4b3ce5fd98ccf18d257cbfbc96ad44
I20260929 19:03:13.573186 14074 leader_election.cc:288] T 00000000000000000000000000000000 P e72376e27ef742698d4b6de6ddf850fa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 19:03:13.573207 14077 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e72376e27ef742698d4b6de6ddf850fa [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:03:13.573316 14077 raft_consensus.cc:686] T 00000000000000000000000000000000 P e72376e27ef742698d4b6de6ddf850fa [term 1 LEADER]: Becoming Leader. State: Replica: e72376e27ef742698d4b6de6ddf850fa, State: Running, Role: LEADER
I20260929 19:03:13.573422 14074 sys_catalog.cc:564] T 00000000000000000000000000000000 P e72376e27ef742698d4b6de6ddf850fa [sys.catalog]: configured and running, proceeding with master startup.
I20260929 19:03:13.573447 14077 consensus_queue.cc:237] T 00000000000000000000000000000000 P e72376e27ef742698d4b6de6ddf850fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e72376e27ef742698d4b6de6ddf850fa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34727 } }
I20260929 19:03:13.573837 14077 sys_catalog.cc:455] T 00000000000000000000000000000000 P e72376e27ef742698d4b6de6ddf850fa [sys.catalog]: SysCatalogTable state changed. Reason: New leader e72376e27ef742698d4b6de6ddf850fa. Latest consensus state: current_term: 1 leader_uuid: "e72376e27ef742698d4b6de6ddf850fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e72376e27ef742698d4b6de6ddf850fa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34727 } } }
I20260929 19:03:13.573992 14077 sys_catalog.cc:458] T 00000000000000000000000000000000 P e72376e27ef742698d4b6de6ddf850fa [sys.catalog]: This master's current role is: LEADER
I20260929 19:03:13.573971 14079 sys_catalog.cc:455] T 00000000000000000000000000000000 P e72376e27ef742698d4b6de6ddf850fa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e72376e27ef742698d4b6de6ddf850fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e72376e27ef742698d4b6de6ddf850fa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34727 } } }
I20260929 19:03:13.574187 14079 sys_catalog.cc:458] T 00000000000000000000000000000000 P e72376e27ef742698d4b6de6ddf850fa [sys.catalog]: This master's current role is: LEADER
I20260929 19:03:13.574244 14088 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 19:03:13.574747 14088 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 19:03:13.576366 14088 catalog_manager.cc:1347] Generated new cluster ID: 3bb9d7bc0af1487f8c79b087607d6fc7
I20260929 19:03:13.576413 14088 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 19:03:13.580544 14088 catalog_manager.cc:1370] Generated new certificate authority record
Sep 29 19:03:13 dist-test-slave-lqkv krb5kdc[13993](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708593, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 19:03:13.581104 14088 catalog_manager.cc:1504] Loading token signing keys...
I20260929 19:03:13.584678 14083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:13.573763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54994 (local address 127.0.98.62:34727)
0929 19:03:13.573943 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:13.573955 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:13.573987 (+ 32us) server_negotiation.cc:407] Connection header received
0929 19:03:13.574274 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:13.574282 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:13.574339 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:13.574448 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:13.575515 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:13.576879 (+ 1364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:13.577901 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:13.578518 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:13.578627 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:13.581266 (+ 2639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:03:13.581270 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:03:13.581270 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:03:13.581282 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:03:13.583145 (+ 1863us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:13.583521 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:13.583524 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:13.583524 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:13.583626 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:13.583800 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:13.583801 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:13.583801 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:13.584040 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:03:13.584188 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:13.584310 (+ 122us) server_negotiation.cc:299] Negotiation successful
0929 19:03:13.584474 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":78,"threads_started":1}
I20260929 19:03:13.585541 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 19:03:13.593181 14088 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e72376e27ef742698d4b6de6ddf850fa: Generated new TSK 0
I20260929 19:03:13.593338 14088 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 19:03:13.639336 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUbnLHE/build/release/bin/kudu with pid 14006
2026-09-29T19:03:13Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1280 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 19:03:13 dist-test-slave-lqkv krb5kdc[14101](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 19:03:13 dist-test-slave-lqkv krb5kdc[14101](info): set up 2 sockets
Sep 29 19:03:13 dist-test-slave-lqkv krb5kdc[14101](info): commencing operation
krb5kdc: starting...
W20260929 19:03:14.648640 392 mini_kdc.cc:121] Time spent starting KDC: real 0.992s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/krb5kdc/test-user.keytab.
Sep 29 19:03:14 dist-test-slave-lqkv krb5kdc[14101](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708594, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T19:03:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T19:03:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 19:03:14.751369 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUbnLHE/build/release/bin/kudu
/tmp/dist-test-taskUbnLHE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34379
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34379
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:03:14.822731 14114 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:03:14.822939 14114 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:03:14.822971 14114 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:03:14.824043 14114 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 19:03:14.824102 14114 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:03:14.824126 14114 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:03:14.824144 14114 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 19:03:14.824173 14114 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 19:03:14.825604 14114 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34379
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34379
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14114
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 18:09:13 UTC on ccycloud.quasar-hpkalf.root.comops.site
build id 4680913
I20260929 19:03:14.825865 14114 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:03:14.826069 14114 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:03:14.830579 14119 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:03:14.830616 14122 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:03:14.830730 14114 server_base.cc:1034] running on GCE node
W20260929 19:03:14.830551 14120 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:03:14.831004 14114 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:03:14.831256 14114 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:03:14.832407 14114 hybrid_clock.cc:648] HybridClock initialized: now 1790708594832401 us; error 54 us; skew 500 ppm
Sep 29 19:03:14 dist-test-slave-lqkv krb5kdc[14101](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708594, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:03:14.840232 14114 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 19:03:14.840648 14114 webserver.cc:460] Webserver started at http://127.0.98.62:33305/ using document root <none> and password file <none>
I20260929 19:03:14.840840 14114 fs_manager.cc:359] Metadata directory not provided
I20260929 19:03:14.840896 14114 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:03:14.840973 14114 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:03:14.841796 14114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/minicluster-data/master-0/data/instance:
uuid: "0c22890aea31494ea792c6d47109b7e0"
format_stamp: "Formatted at 2026-09-29 19:03:14 on dist-test-slave-lqkv"
server_key: "2fc699f4d0b98c0190e7fb5a1d64f91c"
server_key_iv: "3b7ccfe52eebb0b6bf119b828e6880a7"
server_key_version: "encryptionkey@0"
I20260929 19:03:14.842061 14114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/minicluster-data/master-0/wal/instance:
uuid: "0c22890aea31494ea792c6d47109b7e0"
format_stamp: "Formatted at 2026-09-29 19:03:14 on dist-test-slave-lqkv"
server_key: "2fc699f4d0b98c0190e7fb5a1d64f91c"
server_key_iv: "3b7ccfe52eebb0b6bf119b828e6880a7"
server_key_version: "encryptionkey@0"
I20260929 19:03:14.843125 14114 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 19:03:14.843729 14129 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:14.843878 14114 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 19:03:14.843962 14114 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/minicluster-data/master-0/wal
uuid: "0c22890aea31494ea792c6d47109b7e0"
format_stamp: "Formatted at 2026-09-29 19:03:14 on dist-test-slave-lqkv"
server_key: "2fc699f4d0b98c0190e7fb5a1d64f91c"
server_key_iv: "3b7ccfe52eebb0b6bf119b828e6880a7"
server_key_version: "encryptionkey@0"
I20260929 19:03:14.844015 14114 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:03:14.863206 14114 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:03:14.864789 14114 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:03:14.864970 14114 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:03:14.868404 14114 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34379
I20260929 19:03:14.868453 14181 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34379 every 8 connection(s)
I20260929 19:03:14.868733 14114 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/minicluster-data/master-0/data/info.pb
I20260929 19:03:14.869632 14182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:03:14.871482 14182 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c22890aea31494ea792c6d47109b7e0: Bootstrap starting.
I20260929 19:03:14.871968 14182 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0c22890aea31494ea792c6d47109b7e0: Neither blocks nor log segments found. Creating new log.
I20260929 19:03:14.872219 14182 log.cc:824] T 00000000000000000000000000000000 P 0c22890aea31494ea792c6d47109b7e0: Log is configured to *not* fsync() on all Append() calls
I20260929 19:03:14.872793 14182 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c22890aea31494ea792c6d47109b7e0: No bootstrap required, opened a new log
I20260929 19:03:14.874202 14182 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0c22890aea31494ea792c6d47109b7e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c22890aea31494ea792c6d47109b7e0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34379 } }
I20260929 19:03:14.874315 14182 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0c22890aea31494ea792c6d47109b7e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:03:14.874362 14182 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0c22890aea31494ea792c6d47109b7e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c22890aea31494ea792c6d47109b7e0, State: Initialized, Role: FOLLOWER
I20260929 19:03:14.874492 14182 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0c22890aea31494ea792c6d47109b7e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c22890aea31494ea792c6d47109b7e0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34379 } }
I20260929 19:03:14.874565 14182 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0c22890aea31494ea792c6d47109b7e0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 19:03:14.874611 14182 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0c22890aea31494ea792c6d47109b7e0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 19:03:14.874655 14182 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0c22890aea31494ea792c6d47109b7e0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:03:14.875171 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUbnLHE/build/release/bin/kudu as pid 14114
I20260929 19:03:14.875324 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/minicluster-data/master-0/wal/instance
I20260929 19:03:14.875456 14182 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0c22890aea31494ea792c6d47109b7e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c22890aea31494ea792c6d47109b7e0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34379 } }
I20260929 19:03:14.875545 392 external_mini_cluster.cc:1428] Setting key 05ecb3defa93a62bbacdd170374ed336
I20260929 19:03:14.875553 14182 leader_election.cc:302] T 00000000000000000000000000000000 P 0c22890aea31494ea792c6d47109b7e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0c22890aea31494ea792c6d47109b7e0; no voters:
I20260929 19:03:14.875699 14182 leader_election.cc:288] T 00000000000000000000000000000000 P 0c22890aea31494ea792c6d47109b7e0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 19:03:14.875762 14185 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0c22890aea31494ea792c6d47109b7e0 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:03:14.875908 14185 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0c22890aea31494ea792c6d47109b7e0 [term 1 LEADER]: Becoming Leader. State: Replica: 0c22890aea31494ea792c6d47109b7e0, State: Running, Role: LEADER
I20260929 19:03:14.875946 14182 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0c22890aea31494ea792c6d47109b7e0 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 19:03:14.876020 14185 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0c22890aea31494ea792c6d47109b7e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c22890aea31494ea792c6d47109b7e0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34379 } }
I20260929 19:03:14.876428 14186 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c22890aea31494ea792c6d47109b7e0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0c22890aea31494ea792c6d47109b7e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c22890aea31494ea792c6d47109b7e0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34379 } } }
I20260929 19:03:14.876576 14186 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c22890aea31494ea792c6d47109b7e0 [sys.catalog]: This master's current role is: LEADER
I20260929 19:03:14.876478 14187 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c22890aea31494ea792c6d47109b7e0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0c22890aea31494ea792c6d47109b7e0. Latest consensus state: current_term: 1 leader_uuid: "0c22890aea31494ea792c6d47109b7e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c22890aea31494ea792c6d47109b7e0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34379 } } }
I20260929 19:03:14.877015 14187 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c22890aea31494ea792c6d47109b7e0 [sys.catalog]: This master's current role is: LEADER
I20260929 19:03:14.877149 14196 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 19:03:14.877696 14196 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 19:03:14.878965 14196 catalog_manager.cc:1347] Generated new cluster ID: 1267f941b71f420d9709dc2a7f743fb7
I20260929 19:03:14.879022 14196 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 29 19:03:14 dist-test-slave-lqkv krb5kdc[14101](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708594, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 19:03:14.883487 14196 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 19:03:14.884094 14196 catalog_manager.cc:1504] Loading token signing keys...
I20260929 19:03:14.887324 14189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:14.876116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51860 (local address 127.0.98.62:34379)
0929 19:03:14.876679 (+ 563us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:14.876688 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:14.876719 (+ 31us) server_negotiation.cc:407] Connection header received
0929 19:03:14.876821 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:14.876849 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:14.876910 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:14.877071 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:14.877779 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:14.879017 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:14.879944 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:14.880464 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:14.880828 (+ 364us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:14.883026 (+ 2198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:03:14.883030 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:03:14.883030 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:03:14.883040 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:03:14.885476 (+ 2436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:14.885876 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:14.885879 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:14.885879 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:14.885976 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:14.886183 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:14.886184 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:14.886184 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:14.886432 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:03:14.886955 (+ 523us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:14.887016 (+ 61us) server_negotiation.cc:299] Negotiation successful
0929 19:03:14.887139 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":517,"thread_start_us":69,"threads_started":1}
I20260929 19:03:14.889786 14196 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0c22890aea31494ea792c6d47109b7e0: Generated new TSK 0
I20260929 19:03:14.889958 14196 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260929 19:03:14.915724 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUbnLHE/build/release/bin/kudu
/tmp/dist-test-taskUbnLHE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34379
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:03:14.991135 14208 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:03:14.991300 14208 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:03:14.991320 14208 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:03:14.992331 14208 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:03:14.992367 14208 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:03:14.992396 14208 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 19:03:14.993721 14208 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34379
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14208
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 18:09:13 UTC on ccycloud.quasar-hpkalf.root.comops.site
build id 4680913
I20260929 19:03:14.993925 14208 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:03:14.994130 14208 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:03:14.998711 14213 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:03:14.998708 14216 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:03:14.998719 14214 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:03:14.998775 14208 server_base.cc:1034] running on GCE node
I20260929 19:03:14.999086 14208 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:03:14.999320 14208 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:03:15.000463 14208 hybrid_clock.cc:648] HybridClock initialized: now 1790708595000450 us; error 27 us; skew 500 ppm
Sep 29 19:03:14 dist-test-slave-lqkv krb5kdc[14101](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708594, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:03:15.003003 14208 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 19:03:15.003391 14208 webserver.cc:460] Webserver started at http://127.0.98.1:37081/ using document root <none> and password file <none>
I20260929 19:03:15.003594 14208 fs_manager.cc:359] Metadata directory not provided
I20260929 19:03:15.003636 14208 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:03:15.003765 14208 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:03:15.004719 14208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/minicluster-data/ts-0/data/instance:
uuid: "1ec62c6e71024161b5f1f49e6620aa3f"
format_stamp: "Formatted at 2026-09-29 19:03:14 on dist-test-slave-lqkv"
server_key: "93f2f8110ad1d37608b91bf0c68cfad4"
server_key_iv: "67dcdde50b640040b7694967eaa69b39"
server_key_version: "encryptionkey@0"
I20260929 19:03:15.005126 14208 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/minicluster-data/ts-0/wal/instance:
uuid: "1ec62c6e71024161b5f1f49e6620aa3f"
format_stamp: "Formatted at 2026-09-29 19:03:14 on dist-test-slave-lqkv"
server_key: "93f2f8110ad1d37608b91bf0c68cfad4"
server_key_iv: "67dcdde50b640040b7694967eaa69b39"
server_key_version: "encryptionkey@0"
I20260929 19:03:15.006350 14208 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 19:03:15.007014 14223 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:15.007189 14208 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 19:03:15.007253 14208 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/minicluster-data/ts-0/wal
uuid: "1ec62c6e71024161b5f1f49e6620aa3f"
format_stamp: "Formatted at 2026-09-29 19:03:14 on dist-test-slave-lqkv"
server_key: "93f2f8110ad1d37608b91bf0c68cfad4"
server_key_iv: "67dcdde50b640040b7694967eaa69b39"
server_key_version: "encryptionkey@0"
I20260929 19:03:15.007333 14208 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:03:15.027911 14208 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:03:15.029491 14208 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:03:15.029614 14208 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:03:15.029806 14208 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:03:15.030062 14208 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:03:15.030092 14208 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:15.030110 14208 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:03:15.030153 14208 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:15.035733 14208 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38761
I20260929 19:03:15.035784 14336 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38761 every 8 connection(s)
I20260929 19:03:15.036057 14208 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/minicluster-data/ts-0/data/info.pb
I20260929 19:03:15.039853 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUbnLHE/build/release/bin/kudu as pid 14208
I20260929 19:03:15.039990 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/minicluster-data/ts-0/wal/instance
I20260929 19:03:15.040210 392 external_mini_cluster.cc:1428] Setting key b9d8d23b20fbf95c229331daeca6d0fe
Sep 29 19:03:15 dist-test-slave-lqkv krb5kdc[14101](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708594, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 19:03:15.045537 14339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:15.036717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34379 (local address 127.0.98.1:41165)
0929 19:03:15.036896 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:15.036926 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:15.037358 (+ 432us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:15.037598 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:15.037601 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:15.038129 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:03:15.038290 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:15.038294 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:15.038955 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:15.038957 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:15.039765 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:15.039768 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:15.040233 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:15.040234 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:15.040362 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:15.040860 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:03:15.040871 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:03:15.042478 (+ 1607us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:03:15.044277 (+ 1799us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:03:15.044282 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:03:15.044284 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:03:15.044502 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:03:15.044699 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:03:15.044699 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:03:15.044700 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:03:15.044786 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:03:15.045115 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:03:15.045120 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:03:15.045221 (+ 101us) client_negotiation.cc:769] Sending connection context
0929 19:03:15.045292 (+ 71us) client_negotiation.cc:240] Negotiation successful
0929 19:03:15.045398 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":93,"threads_started":1}
I20260929 19:03:15.045624 14189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:15.036737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41165 (local address 127.0.98.62:34379)
0929 19:03:15.036779 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:15.036781 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:15.036997 (+ 216us) server_negotiation.cc:407] Connection header received
0929 19:03:15.037454 (+ 457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:15.037455 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:15.037468 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:15.037526 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:15.038400 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:15.038867 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:15.039871 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:15.040161 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:15.040210 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:15.042569 (+ 2359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:03:15.042570 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:03:15.042570 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:03:15.042577 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:03:15.044175 (+ 1598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:15.044562 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:15.044563 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:15.044564 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:15.044628 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:15.044907 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:15.044908 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:15.044908 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:15.045048 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:03:15.045121 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:15.045374 (+ 253us) server_negotiation.cc:299] Negotiation successful
0929 19:03:15.045485 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260929 19:03:15.046156 14337 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34379
I20260929 19:03:15.046252 14337 heartbeater.cc:464] Registering TS with master...
I20260929 19:03:15.046448 14337 heartbeater.cc:511] Master 127.0.98.62:34379 requested a full tablet report, sending...
I20260929 19:03:15.046947 14146 ts_manager.cc:194] Registered new tserver with Master: 1ec62c6e71024161b5f1f49e6620aa3f (127.0.98.1:38761)
I20260929 19:03:15.047864 14146 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:41165
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260929 19:03:15.065035 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUbnLHE/build/release/bin/kudu
/tmp/dist-test-taskUbnLHE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34379
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:03:15.138330 14342 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:03:15.138556 14342 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:03:15.138581 14342 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:03:15.139577 14342 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:03:15.139613 14342 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:03:15.139636 14342 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 19:03:15.140910 14342 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34379
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14342
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 18:09:13 UTC on ccycloud.quasar-hpkalf.root.comops.site
build id 4680913
I20260929 19:03:15.141052 14342 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:03:15.141212 14342 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:03:15.145468 14350 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:03:15.145468 14348 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:03:15.145498 14347 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:03:15.145713 14342 server_base.cc:1034] running on GCE node
I20260929 19:03:15.145895 14342 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:03:15.146142 14342 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:03:15.147289 14342 hybrid_clock.cc:648] HybridClock initialized: now 1790708595147281 us; error 27 us; skew 500 ppm
Sep 29 19:03:15 dist-test-slave-lqkv krb5kdc[14101](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708595, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:03:15.149504 14342 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 19:03:15.149896 14342 webserver.cc:460] Webserver started at http://127.0.98.2:42663/ using document root <none> and password file <none>
I20260929 19:03:15.150085 14342 fs_manager.cc:359] Metadata directory not provided
I20260929 19:03:15.150123 14342 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:03:15.150244 14342 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:03:15.151185 14342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/minicluster-data/ts-1/data/instance:
uuid: "81a3fe2385c2401cad7e2d8a240a04b0"
format_stamp: "Formatted at 2026-09-29 19:03:15 on dist-test-slave-lqkv"
server_key: "fedbfad63fbe912f1d9b312a62c57d15"
server_key_iv: "4a4e9a95de5686064f3d6f5e4f3a3cb5"
server_key_version: "encryptionkey@0"
I20260929 19:03:15.151508 14342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/minicluster-data/ts-1/wal/instance:
uuid: "81a3fe2385c2401cad7e2d8a240a04b0"
format_stamp: "Formatted at 2026-09-29 19:03:15 on dist-test-slave-lqkv"
server_key: "fedbfad63fbe912f1d9b312a62c57d15"
server_key_iv: "4a4e9a95de5686064f3d6f5e4f3a3cb5"
server_key_version: "encryptionkey@0"
I20260929 19:03:15.152683 14342 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 19:03:15.153388 14357 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:15.153548 14342 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:15.153636 14342 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/minicluster-data/ts-1/wal
uuid: "81a3fe2385c2401cad7e2d8a240a04b0"
format_stamp: "Formatted at 2026-09-29 19:03:15 on dist-test-slave-lqkv"
server_key: "fedbfad63fbe912f1d9b312a62c57d15"
server_key_iv: "4a4e9a95de5686064f3d6f5e4f3a3cb5"
server_key_version: "encryptionkey@0"
I20260929 19:03:15.153713 14342 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:03:15.166122 14342 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:03:15.168207 14342 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:03:15.168347 14342 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:03:15.168551 14342 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:03:15.168795 14342 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:03:15.168874 14342 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:15.168903 14342 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:03:15.168927 14342 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:15.174129 14342 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42897
I20260929 19:03:15.174160 14470 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42897 every 8 connection(s)
I20260929 19:03:15.174432 14342 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/minicluster-data/ts-1/data/info.pb
I20260929 19:03:15.179028 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUbnLHE/build/release/bin/kudu as pid 14342
I20260929 19:03:15.179144 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/minicluster-data/ts-1/wal/instance
I20260929 19:03:15.179329 392 external_mini_cluster.cc:1428] Setting key d4f1d0fc1594bb0537b11b0048ef573f
Sep 29 19:03:15 dist-test-slave-lqkv krb5kdc[14101](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708595, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 19:03:15.182766 14473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:15.174977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34379 (local address 127.0.98.2:37467)
0929 19:03:15.175097 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:15.175124 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:15.175607 (+ 483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:15.175791 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:15.175793 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:15.176355 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:03:15.176547 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:15.176551 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:15.177251 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:15.177251 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:15.177935 (+ 684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:15.177937 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:15.178264 (+ 327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:15.178265 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:15.178375 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:15.178715 (+ 340us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:03:15.178722 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:03:15.180042 (+ 1320us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:03:15.181534 (+ 1492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:03:15.181540 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:03:15.181542 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:03:15.181819 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:03:15.182003 (+ 184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:03:15.182004 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:03:15.182004 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:03:15.182118 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:03:15.182382 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:03:15.182386 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:03:15.182472 (+ 86us) client_negotiation.cc:769] Sending connection context
0929 19:03:15.182538 (+ 66us) client_negotiation.cc:240] Negotiation successful
0929 19:03:15.182646 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":73,"threads_started":1}
I20260929 19:03:15.182801 14189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:15.174990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37467 (local address 127.0.98.62:34379)
0929 19:03:15.175027 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:15.175029 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:15.175217 (+ 188us) server_negotiation.cc:407] Connection header received
0929 19:03:15.175688 (+ 471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:15.175689 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:15.175698 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:15.175747 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:15.176623 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:15.177182 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:15.178003 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:15.178166 (+ 163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:15.178241 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:15.180128 (+ 1887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:03:15.180129 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:03:15.180130 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:03:15.180136 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:03:15.181461 (+ 1325us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:15.181890 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:15.181891 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:15.181891 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:15.181935 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:15.182196 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:15.182197 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:15.182198 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:15.182320 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:03:15.182380 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:15.182577 (+ 197us) server_negotiation.cc:299] Negotiation successful
0929 19:03:15.182675 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260929 19:03:15.183336 14471 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34379
I20260929 19:03:15.183416 14471 heartbeater.cc:464] Registering TS with master...
I20260929 19:03:15.183555 14471 heartbeater.cc:511] Master 127.0.98.62:34379 requested a full tablet report, sending...
I20260929 19:03:15.183871 14146 ts_manager.cc:194] Registered new tserver with Master: 81a3fe2385c2401cad7e2d8a240a04b0 (127.0.98.2:42897)
I20260929 19:03:15.184206 14146 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:37467
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260929 19:03:15.201491 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUbnLHE/build/release/bin/kudu
/tmp/dist-test-taskUbnLHE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34379
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:03:15.275527 14476 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:03:15.275701 14476 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:03:15.275723 14476 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:03:15.277055 14476 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:03:15.277117 14476 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:03:15.277141 14476 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 19:03:15.278373 14476 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34379
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14476
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 18:09:13 UTC on ccycloud.quasar-hpkalf.root.comops.site
build id 4680913
I20260929 19:03:15.278623 14476 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:03:15.278790 14476 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:03:15.283099 14482 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:03:15.283195 14484 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:03:15.283079 14476 server_base.cc:1034] running on GCE node
W20260929 19:03:15.283085 14481 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:03:15.283421 14476 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:03:15.283690 14476 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:03:15.284840 14476 hybrid_clock.cc:648] HybridClock initialized: now 1790708595284820 us; error 33 us; skew 500 ppm
Sep 29 19:03:15 dist-test-slave-lqkv krb5kdc[14101](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708595, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:03:15.287416 14476 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 19:03:15.287847 14476 webserver.cc:460] Webserver started at http://127.0.98.3:37449/ using document root <none> and password file <none>
I20260929 19:03:15.288064 14476 fs_manager.cc:359] Metadata directory not provided
I20260929 19:03:15.288105 14476 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:03:15.288235 14476 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:03:15.289238 14476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/minicluster-data/ts-2/data/instance:
uuid: "a83b42620a78478f93bbddcd97803ca4"
format_stamp: "Formatted at 2026-09-29 19:03:15 on dist-test-slave-lqkv"
server_key: "54ccd58312e082582eb3352115ce149b"
server_key_iv: "6b6aec32819450b6002b1434ecdc0e65"
server_key_version: "encryptionkey@0"
I20260929 19:03:15.289558 14476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/minicluster-data/ts-2/wal/instance:
uuid: "a83b42620a78478f93bbddcd97803ca4"
format_stamp: "Formatted at 2026-09-29 19:03:15 on dist-test-slave-lqkv"
server_key: "54ccd58312e082582eb3352115ce149b"
server_key_iv: "6b6aec32819450b6002b1434ecdc0e65"
server_key_version: "encryptionkey@0"
I20260929 19:03:15.290709 14476 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 19:03:15.291285 14491 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:15.291452 14476 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 19:03:15.291527 14476 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/minicluster-data/ts-2/wal
uuid: "a83b42620a78478f93bbddcd97803ca4"
format_stamp: "Formatted at 2026-09-29 19:03:15 on dist-test-slave-lqkv"
server_key: "54ccd58312e082582eb3352115ce149b"
server_key_iv: "6b6aec32819450b6002b1434ecdc0e65"
server_key_version: "encryptionkey@0"
I20260929 19:03:15.291586 14476 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:03:15.343652 14476 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:03:15.345464 14476 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:03:15.345616 14476 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:03:15.345844 14476 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:03:15.346199 14476 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:03:15.346251 14476 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:15.346292 14476 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:03:15.346319 14476 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:15.351948 14476 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46333
I20260929 19:03:15.352018 14604 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46333 every 8 connection(s)
I20260929 19:03:15.352316 14476 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/minicluster-data/ts-2/data/info.pb
I20260929 19:03:15.355679 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUbnLHE/build/release/bin/kudu as pid 14476
I20260929 19:03:15.355784 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/minicluster-data/ts-2/wal/instance
I20260929 19:03:15.355957 392 external_mini_cluster.cc:1428] Setting key 7ee6ffa938caa87204991f0b3fe43eb1
Sep 29 19:03:15 dist-test-slave-lqkv krb5kdc[14101](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708595, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 19:03:15.360989 14189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:15.353035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37081 (local address 127.0.98.62:34379)
0929 19:03:15.353108 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:15.353110 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:15.353305 (+ 195us) server_negotiation.cc:407] Connection header received
0929 19:03:15.353736 (+ 431us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:15.353736 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:15.353744 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:15.353796 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:15.354602 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:15.355041 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:15.356316 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:15.356617 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:15.356678 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:15.358850 (+ 2172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:03:15.358851 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:03:15.358851 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:03:15.358857 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:03:15.359898 (+ 1041us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:15.360223 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:15.360224 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:15.360224 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:15.360269 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:15.360471 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:15.360471 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:15.360472 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:15.360557 (+ 85us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:03:15.360603 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:15.360856 (+ 253us) server_negotiation.cc:299] Negotiation successful
0929 19:03:15.360931 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260929 19:03:15.361179 14607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:15.353023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34379 (local address 127.0.98.3:37081)
0929 19:03:15.353191 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:15.353228 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:15.353661 (+ 433us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:15.353844 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:15.353847 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:15.354368 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:03:15.354535 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:15.354539 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:15.355110 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:15.355111 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:15.356227 (+ 1116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:15.356230 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:15.356848 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:15.356849 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:15.356999 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:15.357518 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:03:15.357525 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:03:15.358766 (+ 1241us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:03:15.359962 (+ 1196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:03:15.359967 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:03:15.359968 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:03:15.360162 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:03:15.360323 (+ 161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:03:15.360324 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:03:15.360324 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:03:15.360421 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:03:15.360609 (+ 188us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:03:15.360612 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:03:15.360710 (+ 98us) client_negotiation.cc:769] Sending connection context
0929 19:03:15.360783 (+ 73us) client_negotiation.cc:240] Negotiation successful
0929 19:03:15.360956 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":104,"threads_started":1}
I20260929 19:03:15.361613 14605 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34379
I20260929 19:03:15.361699 14605 heartbeater.cc:464] Registering TS with master...
I20260929 19:03:15.361851 14605 heartbeater.cc:511] Master 127.0.98.62:34379 requested a full tablet report, sending...
I20260929 19:03:15.362195 14146 ts_manager.cc:194] Registered new tserver with Master: a83b42620a78478f93bbddcd97803ca4 (127.0.98.3:46333)
I20260929 19:03:15.362507 14146 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:37081
I20260929 19:03:15.367717 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 19:03:15.374580 14189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:15.369061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51868 (local address 127.0.98.62:34379)
0929 19:03:15.369118 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:15.369120 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:15.369191 (+ 71us) server_negotiation.cc:407] Connection header received
0929 19:03:15.369256 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:15.369257 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:15.369266 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:15.369313 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:15.369814 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:15.370276 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:15.371004 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:15.371219 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:15.371288 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:15.372415 (+ 1127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:03:15.372417 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:03:15.372417 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:03:15.372423 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:03:15.373711 (+ 1288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:15.373982 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:15.373983 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:15.373983 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:15.374026 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:15.374160 (+ 134us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:15.374160 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:15.374161 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:15.374263 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:03:15.374328 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:15.374430 (+ 102us) server_negotiation.cc:299] Negotiation successful
0929 19:03:15.374482 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260929 19:03:15.375900 14146 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51868:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 19:03:15.377859 14146 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 19:03:15.384156 14615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:15.379939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46333 (local address 127.0.0.1:37652)
0929 19:03:15.380333 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:15.380366 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:15.380440 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:15.380646 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:15.380648 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:15.380650 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:03:15.380703 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:15.380704 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:15.381985 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:15.381986 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:15.382968 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:15.382970 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:15.383865 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:15.383866 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:15.383962 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:15.383976 (+ 14us) client_negotiation.cc:769] Sending connection context
0929 19:03:15.384038 (+ 62us) client_negotiation.cc:240] Negotiation successful
0929 19:03:15.384090 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":48,"threads_started":1}
I20260929 19:03:15.384258 14613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:15.379961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42897 (local address 127.0.0.1:44828)
0929 19:03:15.380554 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:15.380563 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:15.380720 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:15.381024 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:15.381024 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:15.381026 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:03:15.381064 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:15.381066 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:15.381883 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:15.381884 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:15.383082 (+ 1198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:15.383084 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:15.384008 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:15.384009 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:15.384056 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:15.384059 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 19:03:15.384104 (+ 45us) client_negotiation.cc:240] Negotiation successful
0929 19:03:15.384156 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":584,"mutex_wait_us":38,"thread_start_us":47,"threads_started":1}
I20260929 19:03:15.384418 14616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:15.380005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37652 (local address 127.0.98.3:46333)
0929 19:03:15.380143 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:15.380148 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:15.380456 (+ 308us) server_negotiation.cc:407] Connection header received
0929 19:03:15.380536 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:15.380538 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:15.380554 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:15.380631 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:03:15.380789 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:15.381849 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:15.383061 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:15.383779 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:15.383853 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:15.383908 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:15.384263 (+ 355us) server_negotiation.cc:299] Negotiation successful
0929 19:03:15.384320 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":72,"threads_started":1}
I20260929 19:03:15.384436 14614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:15.379939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38761 (local address 127.0.0.1:46952)
0929 19:03:15.380338 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:15.380367 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:15.380458 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:15.380969 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:15.380969 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:15.380972 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:03:15.381019 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:15.381021 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:15.382243 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:15.382244 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:15.383315 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:15.383317 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:15.383972 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:15.383973 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:15.384038 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:15.384041 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 19:03:15.384104 (+ 63us) client_negotiation.cc:240] Negotiation successful
0929 19:03:15.384157 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":386,"mutex_wait_us":6,"spinlock_wait_cycles":768,"thread_start_us":50,"threads_started":1}
I20260929 19:03:15.384434 14617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:15.380158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44828 (local address 127.0.98.2:42897)
0929 19:03:15.380312 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:15.380316 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:15.380634 (+ 318us) server_negotiation.cc:407] Connection header received
0929 19:03:15.380854 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:15.380857 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:15.380873 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:15.380950 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:03:15.381136 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:15.381786 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:15.383161 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:15.383862 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:15.383933 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:15.383981 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:15.384197 (+ 216us) server_negotiation.cc:299] Negotiation successful
0929 19:03:15.384242 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":89,"threads_started":1}
I20260929 19:03:15.384936 14618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:15.380166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46952 (local address 127.0.98.1:38761)
0929 19:03:15.380321 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:15.380325 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:15.380696 (+ 371us) server_negotiation.cc:407] Connection header received
0929 19:03:15.380747 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:15.380750 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:15.380765 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:15.380890 (+ 125us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:03:15.381103 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:15.382150 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:15.383405 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:15.383902 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:15.383968 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:15.384022 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:15.384118 (+ 96us) server_negotiation.cc:299] Negotiation successful
0929 19:03:15.384155 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":92,"threads_started":1}
I20260929 19:03:15.385087 14539 tablet_service.cc:1467] Processing CreateTablet for tablet e6f87c19fbaf4c97954b20f41afc0d4f (DEFAULT_TABLE table=test-table [id=dbff969460124ba99f978aa90f296380]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:03:15.385110 14405 tablet_service.cc:1467] Processing CreateTablet for tablet e6f87c19fbaf4c97954b20f41afc0d4f (DEFAULT_TABLE table=test-table [id=dbff969460124ba99f978aa90f296380]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:03:15.385437 14405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6f87c19fbaf4c97954b20f41afc0d4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:03:15.385437 14539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6f87c19fbaf4c97954b20f41afc0d4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:03:15.385599 14271 tablet_service.cc:1467] Processing CreateTablet for tablet e6f87c19fbaf4c97954b20f41afc0d4f (DEFAULT_TABLE table=test-table [id=dbff969460124ba99f978aa90f296380]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:03:15.385877 14271 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6f87c19fbaf4c97954b20f41afc0d4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:03:15.387996 14619 tablet_bootstrap.cc:492] T e6f87c19fbaf4c97954b20f41afc0d4f P 81a3fe2385c2401cad7e2d8a240a04b0: Bootstrap starting.
I20260929 19:03:15.388659 14619 tablet_bootstrap.cc:654] T e6f87c19fbaf4c97954b20f41afc0d4f P 81a3fe2385c2401cad7e2d8a240a04b0: Neither blocks nor log segments found. Creating new log.
I20260929 19:03:15.388659 14620 tablet_bootstrap.cc:492] T e6f87c19fbaf4c97954b20f41afc0d4f P a83b42620a78478f93bbddcd97803ca4: Bootstrap starting.
I20260929 19:03:15.388899 14621 tablet_bootstrap.cc:492] T e6f87c19fbaf4c97954b20f41afc0d4f P 1ec62c6e71024161b5f1f49e6620aa3f: Bootstrap starting.
I20260929 19:03:15.389256 14619 log.cc:824] T e6f87c19fbaf4c97954b20f41afc0d4f P 81a3fe2385c2401cad7e2d8a240a04b0: Log is configured to *not* fsync() on all Append() calls
I20260929 19:03:15.389357 14620 tablet_bootstrap.cc:654] T e6f87c19fbaf4c97954b20f41afc0d4f P a83b42620a78478f93bbddcd97803ca4: Neither blocks nor log segments found. Creating new log.
I20260929 19:03:15.389619 14621 tablet_bootstrap.cc:654] T e6f87c19fbaf4c97954b20f41afc0d4f P 1ec62c6e71024161b5f1f49e6620aa3f: Neither blocks nor log segments found. Creating new log.
I20260929 19:03:15.389910 14620 log.cc:824] T e6f87c19fbaf4c97954b20f41afc0d4f P a83b42620a78478f93bbddcd97803ca4: Log is configured to *not* fsync() on all Append() calls
I20260929 19:03:15.390053 14619 tablet_bootstrap.cc:492] T e6f87c19fbaf4c97954b20f41afc0d4f P 81a3fe2385c2401cad7e2d8a240a04b0: No bootstrap required, opened a new log
I20260929 19:03:15.390125 14619 ts_tablet_manager.cc:1397] T e6f87c19fbaf4c97954b20f41afc0d4f P 81a3fe2385c2401cad7e2d8a240a04b0: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 19:03:15.390122 14621 log.cc:824] T e6f87c19fbaf4c97954b20f41afc0d4f P 1ec62c6e71024161b5f1f49e6620aa3f: Log is configured to *not* fsync() on all Append() calls
I20260929 19:03:15.390748 14620 tablet_bootstrap.cc:492] T e6f87c19fbaf4c97954b20f41afc0d4f P a83b42620a78478f93bbddcd97803ca4: No bootstrap required, opened a new log
I20260929 19:03:15.390805 14621 tablet_bootstrap.cc:492] T e6f87c19fbaf4c97954b20f41afc0d4f P 1ec62c6e71024161b5f1f49e6620aa3f: No bootstrap required, opened a new log
I20260929 19:03:15.390816 14620 ts_tablet_manager.cc:1397] T e6f87c19fbaf4c97954b20f41afc0d4f P a83b42620a78478f93bbddcd97803ca4: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 19:03:15.390851 14621 ts_tablet_manager.cc:1397] T e6f87c19fbaf4c97954b20f41afc0d4f P 1ec62c6e71024161b5f1f49e6620aa3f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 19:03:15.391578 14619 raft_consensus.cc:348] T e6f87c19fbaf4c97954b20f41afc0d4f P 81a3fe2385c2401cad7e2d8a240a04b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ec62c6e71024161b5f1f49e6620aa3f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38761 } } peers { permanent_uuid: "a83b42620a78478f93bbddcd97803ca4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46333 } } peers { permanent_uuid: "81a3fe2385c2401cad7e2d8a240a04b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42897 } }
I20260929 19:03:15.391728 14619 raft_consensus.cc:374] T e6f87c19fbaf4c97954b20f41afc0d4f P 81a3fe2385c2401cad7e2d8a240a04b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:03:15.391767 14619 raft_consensus.cc:729] T e6f87c19fbaf4c97954b20f41afc0d4f P 81a3fe2385c2401cad7e2d8a240a04b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81a3fe2385c2401cad7e2d8a240a04b0, State: Initialized, Role: FOLLOWER
I20260929 19:03:15.391889 14619 consensus_queue.cc:260] T e6f87c19fbaf4c97954b20f41afc0d4f P 81a3fe2385c2401cad7e2d8a240a04b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ec62c6e71024161b5f1f49e6620aa3f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38761 } } peers { permanent_uuid: "a83b42620a78478f93bbddcd97803ca4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46333 } } peers { permanent_uuid: "81a3fe2385c2401cad7e2d8a240a04b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42897 } }
I20260929 19:03:15.392151 14619 ts_tablet_manager.cc:1428] T e6f87c19fbaf4c97954b20f41afc0d4f P 81a3fe2385c2401cad7e2d8a240a04b0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 19:03:15.392267 14471 heartbeater.cc:503] Master 127.0.98.62:34379 was elected leader, sending a full tablet report...
I20260929 19:03:15.392644 14620 raft_consensus.cc:348] T e6f87c19fbaf4c97954b20f41afc0d4f P a83b42620a78478f93bbddcd97803ca4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ec62c6e71024161b5f1f49e6620aa3f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38761 } } peers { permanent_uuid: "a83b42620a78478f93bbddcd97803ca4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46333 } } peers { permanent_uuid: "81a3fe2385c2401cad7e2d8a240a04b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42897 } }
I20260929 19:03:15.392827 14620 raft_consensus.cc:374] T e6f87c19fbaf4c97954b20f41afc0d4f P a83b42620a78478f93bbddcd97803ca4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:03:15.392865 14620 raft_consensus.cc:729] T e6f87c19fbaf4c97954b20f41afc0d4f P a83b42620a78478f93bbddcd97803ca4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a83b42620a78478f93bbddcd97803ca4, State: Initialized, Role: FOLLOWER
I20260929 19:03:15.392966 14620 consensus_queue.cc:260] T e6f87c19fbaf4c97954b20f41afc0d4f P a83b42620a78478f93bbddcd97803ca4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ec62c6e71024161b5f1f49e6620aa3f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38761 } } peers { permanent_uuid: "a83b42620a78478f93bbddcd97803ca4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46333 } } peers { permanent_uuid: "81a3fe2385c2401cad7e2d8a240a04b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42897 } }
I20260929 19:03:15.392916 14621 raft_consensus.cc:348] T e6f87c19fbaf4c97954b20f41afc0d4f P 1ec62c6e71024161b5f1f49e6620aa3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ec62c6e71024161b5f1f49e6620aa3f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38761 } } peers { permanent_uuid: "a83b42620a78478f93bbddcd97803ca4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46333 } } peers { permanent_uuid: "81a3fe2385c2401cad7e2d8a240a04b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42897 } }
I20260929 19:03:15.393075 14621 raft_consensus.cc:374] T e6f87c19fbaf4c97954b20f41afc0d4f P 1ec62c6e71024161b5f1f49e6620aa3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:03:15.393121 14621 raft_consensus.cc:729] T e6f87c19fbaf4c97954b20f41afc0d4f P 1ec62c6e71024161b5f1f49e6620aa3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ec62c6e71024161b5f1f49e6620aa3f, State: Initialized, Role: FOLLOWER
I20260929 19:03:15.393218 14620 ts_tablet_manager.cc:1428] T e6f87c19fbaf4c97954b20f41afc0d4f P a83b42620a78478f93bbddcd97803ca4: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20260929 19:03:15.393237 14621 consensus_queue.cc:260] T e6f87c19fbaf4c97954b20f41afc0d4f P 1ec62c6e71024161b5f1f49e6620aa3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ec62c6e71024161b5f1f49e6620aa3f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38761 } } peers { permanent_uuid: "a83b42620a78478f93bbddcd97803ca4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46333 } } peers { permanent_uuid: "81a3fe2385c2401cad7e2d8a240a04b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42897 } }
I20260929 19:03:15.393347 14605 heartbeater.cc:503] Master 127.0.98.62:34379 was elected leader, sending a full tablet report...
I20260929 19:03:15.393482 14621 ts_tablet_manager.cc:1428] T e6f87c19fbaf4c97954b20f41afc0d4f P 1ec62c6e71024161b5f1f49e6620aa3f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 19:03:15.393540 14337 heartbeater.cc:503] Master 127.0.98.62:34379 was elected leader, sending a full tablet report...
W20260929 19:03:15.424918 14472 tablet.cc:2369] T e6f87c19fbaf4c97954b20f41afc0d4f P 81a3fe2385c2401cad7e2d8a240a04b0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 19:03:15.434194 14625 raft_consensus.cc:482] T e6f87c19fbaf4c97954b20f41afc0d4f P 81a3fe2385c2401cad7e2d8a240a04b0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 19:03:15.434280 14625 raft_consensus.cc:504] T e6f87c19fbaf4c97954b20f41afc0d4f P 81a3fe2385c2401cad7e2d8a240a04b0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ec62c6e71024161b5f1f49e6620aa3f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38761 } } peers { permanent_uuid: "a83b42620a78478f93bbddcd97803ca4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46333 } } peers { permanent_uuid: "81a3fe2385c2401cad7e2d8a240a04b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42897 } }
I20260929 19:03:15.434607 14625 leader_election.cc:288] T e6f87c19fbaf4c97954b20f41afc0d4f P 81a3fe2385c2401cad7e2d8a240a04b0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1ec62c6e71024161b5f1f49e6620aa3f (127.0.98.1:38761), a83b42620a78478f93bbddcd97803ca4 (127.0.98.3:46333)
I20260929 19:03:15.438084 14618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:15.434782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54645 (local address 127.0.98.1:38761)
0929 19:03:15.434815 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:15.434817 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:15.434825 (+ 8us) server_negotiation.cc:407] Connection header received
0929 19:03:15.434897 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:15.434899 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:15.434913 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:15.434973 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:03:15.435109 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:15.435841 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:15.437275 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:15.437768 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:15.437839 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:15.437848 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:15.437997 (+ 149us) server_negotiation.cc:299] Negotiation successful
0929 19:03:15.438029 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260929 19:03:15.438084 14616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:15.434816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55061 (local address 127.0.98.3:46333)
0929 19:03:15.434862 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:15.434864 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:15.434964 (+ 100us) server_negotiation.cc:407] Connection header received
0929 19:03:15.435021 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:15.435022 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:15.435029 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:15.435065 (+ 36us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:03:15.435182 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:15.435857 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:15.437277 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:15.437768 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:15.437839 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:15.437848 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:15.438001 (+ 153us) server_negotiation.cc:299] Negotiation successful
0929 19:03:15.438032 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260929 19:03:15.438092 14473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:15.434719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38761 (local address 127.0.98.2:54645)
0929 19:03:15.434767 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:15.434777 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:15.434816 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:15.434987 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:15.434988 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:15.434989 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:03:15.435048 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:15.435050 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:15.435930 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:15.435931 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:15.437200 (+ 1269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:15.437203 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:15.437846 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:15.437847 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:15.437922 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:15.437928 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 19:03:15.437979 (+ 51us) client_negotiation.cc:240] Negotiation successful
0929 19:03:15.438022 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31,"mutex_wait_us":77}
I20260929 19:03:15.438382 14628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:15.434760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46333 (local address 127.0.98.2:55061)
0929 19:03:15.434897 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:15.434905 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:15.434971 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:15.435069 (+ 98us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:15.435069 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:15.435071 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:03:15.435126 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:15.435127 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:15.435946 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:15.435947 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:15.437200 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:15.437203 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:15.437846 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:15.437847 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:15.437922 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:15.437928 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 19:03:15.437979 (+ 51us) client_negotiation.cc:240] Negotiation successful
0929 19:03:15.438317 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"mutex_wait_us":45,"thread_start_us":79,"threads_started":1}
I20260929 19:03:15.438361 14291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6f87c19fbaf4c97954b20f41afc0d4f" candidate_uuid: "81a3fe2385c2401cad7e2d8a240a04b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ec62c6e71024161b5f1f49e6620aa3f" is_pre_election: true
I20260929 19:03:15.438494 14291 raft_consensus.cc:2454] T e6f87c19fbaf4c97954b20f41afc0d4f P 1ec62c6e71024161b5f1f49e6620aa3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 81a3fe2385c2401cad7e2d8a240a04b0 in term 0.
I20260929 19:03:15.438656 14359 leader_election.cc:302] T e6f87c19fbaf4c97954b20f41afc0d4f P 81a3fe2385c2401cad7e2d8a240a04b0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ec62c6e71024161b5f1f49e6620aa3f, 81a3fe2385c2401cad7e2d8a240a04b0; no voters:
I20260929 19:03:15.438671 14559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6f87c19fbaf4c97954b20f41afc0d4f" candidate_uuid: "81a3fe2385c2401cad7e2d8a240a04b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a83b42620a78478f93bbddcd97803ca4" is_pre_election: true
I20260929 19:03:15.438776 14559 raft_consensus.cc:2454] T e6f87c19fbaf4c97954b20f41afc0d4f P a83b42620a78478f93bbddcd97803ca4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 81a3fe2385c2401cad7e2d8a240a04b0 in term 0.
I20260929 19:03:15.438799 14625 raft_consensus.cc:2781] T e6f87c19fbaf4c97954b20f41afc0d4f P 81a3fe2385c2401cad7e2d8a240a04b0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 19:03:15.438845 14625 raft_consensus.cc:482] T e6f87c19fbaf4c97954b20f41afc0d4f P 81a3fe2385c2401cad7e2d8a240a04b0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 19:03:15.438871 14625 raft_consensus.cc:3037] T e6f87c19fbaf4c97954b20f41afc0d4f P 81a3fe2385c2401cad7e2d8a240a04b0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:03:15.439481 14625 raft_consensus.cc:504] T e6f87c19fbaf4c97954b20f41afc0d4f P 81a3fe2385c2401cad7e2d8a240a04b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ec62c6e71024161b5f1f49e6620aa3f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38761 } } peers { permanent_uuid: "a83b42620a78478f93bbddcd97803ca4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46333 } } peers { permanent_uuid: "81a3fe2385c2401cad7e2d8a240a04b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42897 } }
I20260929 19:03:15.439621 14625 leader_election.cc:288] T e6f87c19fbaf4c97954b20f41afc0d4f P 81a3fe2385c2401cad7e2d8a240a04b0 [CANDIDATE]: Term 1 election: Requested vote from peers 1ec62c6e71024161b5f1f49e6620aa3f (127.0.98.1:38761), a83b42620a78478f93bbddcd97803ca4 (127.0.98.3:46333)
I20260929 19:03:15.439754 14291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6f87c19fbaf4c97954b20f41afc0d4f" candidate_uuid: "81a3fe2385c2401cad7e2d8a240a04b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ec62c6e71024161b5f1f49e6620aa3f"
I20260929 19:03:15.439767 14559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6f87c19fbaf4c97954b20f41afc0d4f" candidate_uuid: "81a3fe2385c2401cad7e2d8a240a04b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a83b42620a78478f93bbddcd97803ca4"
I20260929 19:03:15.439821 14291 raft_consensus.cc:3037] T e6f87c19fbaf4c97954b20f41afc0d4f P 1ec62c6e71024161b5f1f49e6620aa3f [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:03:15.439854 14559 raft_consensus.cc:3037] T e6f87c19fbaf4c97954b20f41afc0d4f P a83b42620a78478f93bbddcd97803ca4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:03:15.440496 14291 raft_consensus.cc:2454] T e6f87c19fbaf4c97954b20f41afc0d4f P 1ec62c6e71024161b5f1f49e6620aa3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 81a3fe2385c2401cad7e2d8a240a04b0 in term 1.
I20260929 19:03:15.440500 14559 raft_consensus.cc:2454] T e6f87c19fbaf4c97954b20f41afc0d4f P a83b42620a78478f93bbddcd97803ca4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 81a3fe2385c2401cad7e2d8a240a04b0 in term 1.
I20260929 19:03:15.440642 14359 leader_election.cc:302] T e6f87c19fbaf4c97954b20f41afc0d4f P 81a3fe2385c2401cad7e2d8a240a04b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ec62c6e71024161b5f1f49e6620aa3f, 81a3fe2385c2401cad7e2d8a240a04b0; no voters:
I20260929 19:03:15.440726 14625 raft_consensus.cc:2781] T e6f87c19fbaf4c97954b20f41afc0d4f P 81a3fe2385c2401cad7e2d8a240a04b0 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:03:15.440872 14625 raft_consensus.cc:686] T e6f87c19fbaf4c97954b20f41afc0d4f P 81a3fe2385c2401cad7e2d8a240a04b0 [term 1 LEADER]: Becoming Leader. State: Replica: 81a3fe2385c2401cad7e2d8a240a04b0, State: Running, Role: LEADER
I20260929 19:03:15.440979 14625 consensus_queue.cc:237] T e6f87c19fbaf4c97954b20f41afc0d4f P 81a3fe2385c2401cad7e2d8a240a04b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ec62c6e71024161b5f1f49e6620aa3f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38761 } } peers { permanent_uuid: "a83b42620a78478f93bbddcd97803ca4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46333 } } peers { permanent_uuid: "81a3fe2385c2401cad7e2d8a240a04b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42897 } }
I20260929 19:03:15.441675 14145 catalog_manager.cc:5495] T e6f87c19fbaf4c97954b20f41afc0d4f P 81a3fe2385c2401cad7e2d8a240a04b0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 81a3fe2385c2401cad7e2d8a240a04b0 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "81a3fe2385c2401cad7e2d8a240a04b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ec62c6e71024161b5f1f49e6620aa3f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a83b42620a78478f93bbddcd97803ca4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46333 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "81a3fe2385c2401cad7e2d8a240a04b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42897 } health_report { overall_health: HEALTHY } } }
I20260929 19:03:15.456864 14617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:15.454755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44840 (local address 127.0.98.2:42897)
0929 19:03:15.454802 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:15.454803 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:15.454811 (+ 8us) server_negotiation.cc:407] Connection header received
0929 19:03:15.454866 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:15.454867 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:15.454881 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:15.454944 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 19:03:15.455049 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:15.455457 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:15.456233 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:15.456408 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:15.456457 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:15.456592 (+ 135us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 19:03:15.456669 (+ 77us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 19:03:15.456721 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:15.456754 (+ 33us) server_negotiation.cc:299] Negotiation successful
0929 19:03:15.456768 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260929 19:03:15.458249 14291 raft_consensus.cc:1261] T e6f87c19fbaf4c97954b20f41afc0d4f P 1ec62c6e71024161b5f1f49e6620aa3f [term 1 FOLLOWER]: Refusing update from remote peer 81a3fe2385c2401cad7e2d8a240a04b0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 19:03:15.458283 14559 raft_consensus.cc:1261] T e6f87c19fbaf4c97954b20f41afc0d4f P a83b42620a78478f93bbddcd97803ca4 [term 1 FOLLOWER]: Refusing update from remote peer 81a3fe2385c2401cad7e2d8a240a04b0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 19:03:15.458607 14625 consensus_queue.cc:1035] T e6f87c19fbaf4c97954b20f41afc0d4f P 81a3fe2385c2401cad7e2d8a240a04b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a83b42620a78478f93bbddcd97803ca4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46333 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 19:03:15.458743 14629 consensus_queue.cc:1035] T e6f87c19fbaf4c97954b20f41afc0d4f P 81a3fe2385c2401cad7e2d8a240a04b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ec62c6e71024161b5f1f49e6620aa3f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 19:03:15.461568 14633 mvcc.cc:205] Tried to move back new op lower bound from 7334742406994792448 to 7334742406926761984. Current Snapshot: MvccSnapshot[applied={T|T < 7334742406994792448 or (T in {7334742406994792448})}]
I20260929 19:03:15.463138 14146 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51868:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 19:03:15.463228 14146 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51868:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 19:03:15.463830 14146 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0c22890aea31494ea792c6d47109b7e0: Sending DeleteTablet for 3 replicas of tablet e6f87c19fbaf4c97954b20f41afc0d4f
I20260929 19:03:15.464143 14405 tablet_service.cc:1514] Processing DeleteTablet for tablet e6f87c19fbaf4c97954b20f41afc0d4f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:03:15 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44828
I20260929 19:03:15.464169 14271 tablet_service.cc:1514] Processing DeleteTablet for tablet e6f87c19fbaf4c97954b20f41afc0d4f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:03:15 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46952
I20260929 19:03:15.464160 14539 tablet_service.cc:1514] Processing DeleteTablet for tablet e6f87c19fbaf4c97954b20f41afc0d4f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:03:15 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37652
I20260929 19:03:15.464424 14644 tablet_replica.cc:330] T e6f87c19fbaf4c97954b20f41afc0d4f P 1ec62c6e71024161b5f1f49e6620aa3f: stopping tablet replica
I20260929 19:03:15.464426 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUbnLHE/build/release/bin/kudu with pid 14208
I20260929 19:03:15.464509 14644 raft_consensus.cc:2229] T e6f87c19fbaf4c97954b20f41afc0d4f P 1ec62c6e71024161b5f1f49e6620aa3f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 19:03:15.465101 14643 tablet_replica.cc:330] T e6f87c19fbaf4c97954b20f41afc0d4f P a83b42620a78478f93bbddcd97803ca4: stopping tablet replica
I20260929 19:03:15.465188 14643 raft_consensus.cc:2229] T e6f87c19fbaf4c97954b20f41afc0d4f P a83b42620a78478f93bbddcd97803ca4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 19:03:15.465278 14643 raft_consensus.cc:2258] T e6f87c19fbaf4c97954b20f41afc0d4f P a83b42620a78478f93bbddcd97803ca4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 19:03:15.465483 14642 tablet_replica.cc:330] T e6f87c19fbaf4c97954b20f41afc0d4f P 81a3fe2385c2401cad7e2d8a240a04b0: stopping tablet replica
I20260929 19:03:15.465570 14642 raft_consensus.cc:2229] T e6f87c19fbaf4c97954b20f41afc0d4f P 81a3fe2385c2401cad7e2d8a240a04b0 [term 1 LEADER]: Raft consensus shutting down.
I20260929 19:03:15.465649 14642 raft_consensus.cc:2258] T e6f87c19fbaf4c97954b20f41afc0d4f P 81a3fe2385c2401cad7e2d8a240a04b0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 19:03:15.465994 14643 ts_tablet_manager.cc:1905] T e6f87c19fbaf4c97954b20f41afc0d4f P a83b42620a78478f93bbddcd97803ca4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 19:03:15.467312 14642 ts_tablet_manager.cc:1905] T e6f87c19fbaf4c97954b20f41afc0d4f P 81a3fe2385c2401cad7e2d8a240a04b0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 19:03:15.467521 14643 ts_tablet_manager.cc:1918] T e6f87c19fbaf4c97954b20f41afc0d4f P a83b42620a78478f93bbddcd97803ca4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 19:03:15.467633 14643 log.cc:1196] T e6f87c19fbaf4c97954b20f41afc0d4f P a83b42620a78478f93bbddcd97803ca4: Deleting WAL directory at /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/minicluster-data/ts-2/wal/wals/e6f87c19fbaf4c97954b20f41afc0d4f
I20260929 19:03:15.467996 14643 ts_tablet_manager.cc:1939] T e6f87c19fbaf4c97954b20f41afc0d4f P a83b42620a78478f93bbddcd97803ca4: Deleting consensus metadata
I20260929 19:03:15.468411 14132 catalog_manager.cc:4841] TS a83b42620a78478f93bbddcd97803ca4 (127.0.98.3:46333): tablet e6f87c19fbaf4c97954b20f41afc0d4f (table test-table [id=dbff969460124ba99f978aa90f296380]) successfully deleted
I20260929 19:03:15.468997 14642 ts_tablet_manager.cc:1918] T e6f87c19fbaf4c97954b20f41afc0d4f P 81a3fe2385c2401cad7e2d8a240a04b0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 19:03:15.469074 14642 log.cc:1196] T e6f87c19fbaf4c97954b20f41afc0d4f P 81a3fe2385c2401cad7e2d8a240a04b0: Deleting WAL directory at /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/minicluster-data/ts-1/wal/wals/e6f87c19fbaf4c97954b20f41afc0d4f
I20260929 19:03:15.469310 14642 ts_tablet_manager.cc:1939] T e6f87c19fbaf4c97954b20f41afc0d4f P 81a3fe2385c2401cad7e2d8a240a04b0: Deleting consensus metadata
I20260929 19:03:15.469660 14133 catalog_manager.cc:4841] TS 81a3fe2385c2401cad7e2d8a240a04b0 (127.0.98.2:42897): tablet e6f87c19fbaf4c97954b20f41afc0d4f (table test-table [id=dbff969460124ba99f978aa90f296380]) successfully deleted
W20260929 19:03:15.470175 14131 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:38761 (error 108)
I20260929 19:03:15.470590 14614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:15.470477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38761 (local address 127.0.0.1:46966)
0929 19:03:15.470516 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:15.470559 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38761: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
I20260929 19:03:15.470676 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUbnLHE/build/release/bin/kudu with pid 14342
W20260929 19:03:15.470685 14131 catalog_manager.cc:4584] TS 1ec62c6e71024161b5f1f49e6620aa3f (127.0.98.1:38761): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e6f87c19fbaf4c97954b20f41afc0d4f: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38761: connect: Connection refused (error 111)
I20260929 19:03:15.474931 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUbnLHE/build/release/bin/kudu with pid 14476
I20260929 19:03:15.479689 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUbnLHE/build/release/bin/kudu with pid 14114
I20260929 19:03:15.484313 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUbnLHE/build/release/bin/kudu
/tmp/dist-test-taskUbnLHE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34379
--webserver_interface=127.0.98.62
--webserver_port=33305
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34379
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:03:15.558037 14646 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:03:15.558287 14646 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:03:15.558326 14646 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:03:15.559458 14646 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 19:03:15.559522 14646 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:03:15.559545 14646 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:03:15.559573 14646 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 19:03:15.559603 14646 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 19:03:15.561026 14646 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34379
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34379
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=33305
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14646
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 18:09:13 UTC on ccycloud.quasar-hpkalf.root.comops.site
build id 4680913
I20260929 19:03:15.561286 14646 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:03:15.561496 14646 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:03:15.569434 14652 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:03:15.569443 14655 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:03:15.569443 14653 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:03:15.570039 14646 server_base.cc:1034] running on GCE node
I20260929 19:03:15.570252 14646 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:03:15.570490 14646 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:03:15.571659 14646 hybrid_clock.cc:648] HybridClock initialized: now 1790708595571647 us; error 30 us; skew 500 ppm
Sep 29 19:03:15 dist-test-slave-lqkv krb5kdc[14101](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708595, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:03:15.574033 14646 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 19:03:15.574424 14646 webserver.cc:460] Webserver started at http://127.0.98.62:33305/ using document root <none> and password file <none>
I20260929 19:03:15.574652 14646 fs_manager.cc:359] Metadata directory not provided
I20260929 19:03:15.574700 14646 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:03:15.575980 14646 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 19:03:15.576604 14662 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:15.576762 14646 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 19:03:15.576901 14646 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/minicluster-data/master-0/wal
uuid: "0c22890aea31494ea792c6d47109b7e0"
format_stamp: "Formatted at 2026-09-29 19:03:14 on dist-test-slave-lqkv"
server_key: "2fc699f4d0b98c0190e7fb5a1d64f91c"
server_key_iv: "3b7ccfe52eebb0b6bf119b828e6880a7"
server_key_version: "encryptionkey@0"
I20260929 19:03:15.577222 14646 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:03:15.583964 14646 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:03:15.585744 14646 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:03:15.585896 14646 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:03:15.589457 14646 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34379
I20260929 19:03:15.589483 14714 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34379 every 8 connection(s)
I20260929 19:03:15.589813 14646 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/minicluster-data/master-0/data/info.pb
I20260929 19:03:15.591125 14715 sys_catalog.cc:263] Verifying existing consensus state
I20260929 19:03:15.591625 14715 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c22890aea31494ea792c6d47109b7e0: Bootstrap starting.
I20260929 19:03:15.598567 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUbnLHE/build/release/bin/kudu as pid 14646
I20260929 19:03:15.598932 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUbnLHE/build/release/bin/kudu
/tmp/dist-test-taskUbnLHE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:38761
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:34379
--webserver_port=37081
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 19:03:15.599682 14715 log.cc:824] T 00000000000000000000000000000000 P 0c22890aea31494ea792c6d47109b7e0: Log is configured to *not* fsync() on all Append() calls
I20260929 19:03:15.601575 14715 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c22890aea31494ea792c6d47109b7e0: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 19:03:15.601842 14715 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c22890aea31494ea792c6d47109b7e0: Bootstrap complete.
I20260929 19:03:15.603816 14715 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0c22890aea31494ea792c6d47109b7e0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c22890aea31494ea792c6d47109b7e0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34379 } }
I20260929 19:03:15.604012 14715 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0c22890aea31494ea792c6d47109b7e0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c22890aea31494ea792c6d47109b7e0, State: Initialized, Role: FOLLOWER
I20260929 19:03:15.604157 14715 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0c22890aea31494ea792c6d47109b7e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c22890aea31494ea792c6d47109b7e0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34379 } }
I20260929 19:03:15.604208 14715 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0c22890aea31494ea792c6d47109b7e0 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 19:03:15.604262 14715 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0c22890aea31494ea792c6d47109b7e0 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 19:03:15.604322 14715 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0c22890aea31494ea792c6d47109b7e0 [term 1 FOLLOWER]: Advancing to term 2
I20260929 19:03:15.605347 14715 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0c22890aea31494ea792c6d47109b7e0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c22890aea31494ea792c6d47109b7e0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34379 } }
I20260929 19:03:15.605429 14715 leader_election.cc:302] T 00000000000000000000000000000000 P 0c22890aea31494ea792c6d47109b7e0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0c22890aea31494ea792c6d47109b7e0; no voters:
I20260929 19:03:15.605629 14715 leader_election.cc:288] T 00000000000000000000000000000000 P 0c22890aea31494ea792c6d47109b7e0 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 19:03:15.605660 14719 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0c22890aea31494ea792c6d47109b7e0 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 19:03:15.605773 14719 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0c22890aea31494ea792c6d47109b7e0 [term 2 LEADER]: Becoming Leader. State: Replica: 0c22890aea31494ea792c6d47109b7e0, State: Running, Role: LEADER
I20260929 19:03:15.605876 14715 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0c22890aea31494ea792c6d47109b7e0 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 19:03:15.605854 14719 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0c22890aea31494ea792c6d47109b7e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c22890aea31494ea792c6d47109b7e0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34379 } }
I20260929 19:03:15.606195 14719 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c22890aea31494ea792c6d47109b7e0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0c22890aea31494ea792c6d47109b7e0. Latest consensus state: current_term: 2 leader_uuid: "0c22890aea31494ea792c6d47109b7e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c22890aea31494ea792c6d47109b7e0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34379 } } }
I20260929 19:03:15.606277 14719 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c22890aea31494ea792c6d47109b7e0 [sys.catalog]: This master's current role is: LEADER
I20260929 19:03:15.606197 14720 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c22890aea31494ea792c6d47109b7e0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "0c22890aea31494ea792c6d47109b7e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c22890aea31494ea792c6d47109b7e0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34379 } } }
I20260929 19:03:15.606458 14720 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c22890aea31494ea792c6d47109b7e0 [sys.catalog]: This master's current role is: LEADER
I20260929 19:03:15.607596 14735 catalog_manager.cc:1259] Loaded cluster ID: 1267f941b71f420d9709dc2a7f743fb7
I20260929 19:03:15.607643 14735 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0c22890aea31494ea792c6d47109b7e0: loading cluster ID for follower catalog manager: success
W20260929 19:03:15.607846 14735 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 0c22890aea31494ea792c6d47109b7e0: acquiring CA information for follower catalog manager: Runtime error: could not decrypt private key with the password returned by the configured command: unable to load data from memory: error reading private key: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D07803A:asn1 encoding routines:ASN1_ITEM_EX_D2I:nested asn1 error:tasn_dec.c:386:Type=X509_ALGOR error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=algor, Type=X509_SIG
W20260929 19:03:15.607903 14735 catalog_manager.cc:873] Runtime error: could not decrypt private key with the password returned by the configured command: unable to load data from memory: error reading private key: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D07803A:asn1 encoding routines:ASN1_ITEM_EX_D2I:nested asn1 error:tasn_dec.c:386:Type=X509_ALGOR error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=algor, Type=X509_SIG: failed to prepare follower catalog manager, will retry
I20260929 19:03:15.608923 14725 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 19:03:15.609269 14725 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 19:03:15.609347 14725 catalog_manager.cc:1259] Loaded cluster ID: 1267f941b71f420d9709dc2a7f743fb7
I20260929 19:03:15.609377 14725 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 19:03:15.609553 14725 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260929 19:03:15.613775 14725 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260929 19:03:15.615953 14725 catalog_manager.cc:1504] Loading token signing keys...
I20260929 19:03:15.616122 14725 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 0c22890aea31494ea792c6d47109b7e0: Loaded TSK: 0
I20260929 19:03:15.616317 14725 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 19:03:15.682237 14718 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:03:15.682408 14718 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:03:15.682428 14718 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:03:15.683492 14718 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:03:15.683527 14718 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:03:15.683550 14718 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 19:03:15.685019 14718 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:38761
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=37081
--tserver_master_addrs=127.0.98.62:34379
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14718
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 18:09:13 UTC on ccycloud.quasar-hpkalf.root.comops.site
build id 4680913
I20260929 19:03:15.685226 14718 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:03:15.685459 14718 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:03:15.691030 14742 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:03:15.691045 14743 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:03:15.691344 14718 server_base.cc:1034] running on GCE node
W20260929 19:03:15.691045 14745 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:03:15.692090 14718 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:03:15.692312 14718 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:03:15.693471 14718 hybrid_clock.cc:648] HybridClock initialized: now 1790708595693464 us; error 33 us; skew 500 ppm
Sep 29 19:03:15 dist-test-slave-lqkv krb5kdc[14101](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708595, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:03:15.695787 14718 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 19:03:15.696223 14718 webserver.cc:460] Webserver started at http://127.0.98.1:37081/ using document root <none> and password file <none>
I20260929 19:03:15.696466 14718 fs_manager.cc:359] Metadata directory not provided
I20260929 19:03:15.696519 14718 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:03:15.697873 14718 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 19:03:15.698539 14752 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:15.698714 14718 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 19:03:15.698797 14718 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/minicluster-data/ts-0/wal
uuid: "1ec62c6e71024161b5f1f49e6620aa3f"
format_stamp: "Formatted at 2026-09-29 19:03:14 on dist-test-slave-lqkv"
server_key: "93f2f8110ad1d37608b91bf0c68cfad4"
server_key_iv: "67dcdde50b640040b7694967eaa69b39"
server_key_version: "encryptionkey@0"
I20260929 19:03:15.699102 14718 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:03:15.726514 14718 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:03:15.728492 14718 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:03:15.728645 14718 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:03:15.728930 14718 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:03:15.729354 14759 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260929 19:03:15.731024 14718 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260929 19:03:15.731086 14718 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260929 19:03:15.731127 14718 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260929 19:03:15.731716 14718 ts_tablet_manager.cc:610] Registered 1 tablets
I20260929 19:03:15.731771 14718 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260929 19:03:15.731850 14759 tablet_bootstrap.cc:492] T e6f87c19fbaf4c97954b20f41afc0d4f P 1ec62c6e71024161b5f1f49e6620aa3f: Bootstrap starting.
I20260929 19:03:15.737233 14718 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38761
I20260929 19:03:15.737313 14866 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38761 every 8 connection(s)
I20260929 19:03:15.737596 14718 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/minicluster-data/ts-0/data/info.pb
I20260929 19:03:15.742518 14759 log.cc:824] T e6f87c19fbaf4c97954b20f41afc0d4f P 1ec62c6e71024161b5f1f49e6620aa3f: Log is configured to *not* fsync() on all Append() calls
I20260929 19:03:15.743916 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUbnLHE/build/release/bin/kudu as pid 14718
I20260929 19:03:15.744246 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUbnLHE/build/release/bin/kudu
/tmp/dist-test-taskUbnLHE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:42897
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:34379
--webserver_port=42663
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 19:03:15.744707 14759 tablet_bootstrap.cc:492] T e6f87c19fbaf4c97954b20f41afc0d4f P 1ec62c6e71024161b5f1f49e6620aa3f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 19:03:15.745035 14759 tablet_bootstrap.cc:492] T e6f87c19fbaf4c97954b20f41afc0d4f P 1ec62c6e71024161b5f1f49e6620aa3f: Bootstrap complete.
Sep 29 19:03:15 dist-test-slave-lqkv krb5kdc[14101](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708595, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 19:03:15.746147 14759 ts_tablet_manager.cc:1397] T e6f87c19fbaf4c97954b20f41afc0d4f P 1ec62c6e71024161b5f1f49e6620aa3f: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.006s
I20260929 19:03:15.747347 14759 raft_consensus.cc:348] T e6f87c19fbaf4c97954b20f41afc0d4f P 1ec62c6e71024161b5f1f49e6620aa3f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ec62c6e71024161b5f1f49e6620aa3f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38761 } } peers { permanent_uuid: "a83b42620a78478f93bbddcd97803ca4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46333 } } peers { permanent_uuid: "81a3fe2385c2401cad7e2d8a240a04b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42897 } }
I20260929 19:03:15.747597 14759 raft_consensus.cc:729] T e6f87c19fbaf4c97954b20f41afc0d4f P 1ec62c6e71024161b5f1f49e6620aa3f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ec62c6e71024161b5f1f49e6620aa3f, State: Initialized, Role: FOLLOWER
I20260929 19:03:15.747735 14759 consensus_queue.cc:260] T e6f87c19fbaf4c97954b20f41afc0d4f P 1ec62c6e71024161b5f1f49e6620aa3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ec62c6e71024161b5f1f49e6620aa3f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38761 } } peers { permanent_uuid: "a83b42620a78478f93bbddcd97803ca4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46333 } } peers { permanent_uuid: "81a3fe2385c2401cad7e2d8a240a04b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42897 } }
I20260929 19:03:15.748014 14759 ts_tablet_manager.cc:1428] T e6f87c19fbaf4c97954b20f41afc0d4f P 1ec62c6e71024161b5f1f49e6620aa3f: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260929 19:03:15.749621 14870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:15.738308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48311 (local address 127.0.98.62:34379)
0929 19:03:15.739446 (+ 1138us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:15.739458 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:15.739508 (+ 50us) server_negotiation.cc:407] Connection header received
0929 19:03:15.739646 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:15.739653 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:15.739714 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:15.739899 (+ 185us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:15.740427 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:15.741956 (+ 1529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:15.743019 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:15.743384 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:15.743516 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:15.746133 (+ 2617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:03:15.746137 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:03:15.746137 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:03:15.746150 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:03:15.747935 (+ 1785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:15.748417 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:15.748419 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:15.748419 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:15.748496 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:15.748750 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:15.748750 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:15.748751 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:15.748986 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:03:15.749044 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:15.749354 (+ 310us) server_negotiation.cc:299] Negotiation successful
0929 19:03:15.749503 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1084,"thread_start_us":95,"threads_started":1}
I20260929 19:03:15.749707 14869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:15.738238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34379 (local address 127.0.98.1:48311)
0929 19:03:15.738428 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:15.738533 (+ 105us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:15.739236 (+ 703us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:15.739914 (+ 678us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:15.739917 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:15.740142 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:03:15.740308 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:15.740312 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:15.742093 (+ 1781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:15.742094 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:15.742916 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:15.742919 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:15.743512 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:15.743514 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:15.743635 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:15.744225 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:03:15.744236 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:03:15.746028 (+ 1792us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:03:15.748019 (+ 1991us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:03:15.748024 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:03:15.748026 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:03:15.748329 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:03:15.748557 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:03:15.748558 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:03:15.748558 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:03:15.748689 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:03:15.749050 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:03:15.749057 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:03:15.749206 (+ 149us) client_negotiation.cc:769] Sending connection context
0929 19:03:15.749316 (+ 110us) client_negotiation.cc:240] Negotiation successful
0929 19:03:15.749480 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":89,"threads_started":1}
I20260929 19:03:15.750372 14867 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34379
I20260929 19:03:15.750465 14867 heartbeater.cc:464] Registering TS with master...
I20260929 19:03:15.750635 14867 heartbeater.cc:511] Master 127.0.98.62:34379 requested a full tablet report, sending...
I20260929 19:03:15.751348 14679 ts_manager.cc:194] Registered new tserver with Master: 1ec62c6e71024161b5f1f49e6620aa3f (127.0.98.1:38761)
I20260929 19:03:15.752523 14679 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:48311
I20260929 19:03:15.753786 14867 heartbeater.cc:503] Master 127.0.98.62:34379 was elected leader, sending a full tablet report...
Sep 29 19:03:15 dist-test-slave-lqkv krb5kdc[14101](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708595, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260929 19:03:15.759408 14874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:15.751907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38761 (local address 127.0.0.1:46982)
0929 19:03:15.752134 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:15.752155 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:15.752235 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:15.752391 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:15.752395 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:15.752691 (+ 296us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:03:15.752737 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:15.752741 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:15.753691 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:15.753692 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:15.754473 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:15.754475 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:15.754906 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:15.754910 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:15.754977 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:15.755429 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:03:15.755433 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:03:15.756922 (+ 1489us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:03:15.758352 (+ 1430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:03:15.758358 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:03:15.758360 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:03:15.758542 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:03:15.758773 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:03:15.758774 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:03:15.758774 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:03:15.758825 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:03:15.759085 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:03:15.759090 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:03:15.759133 (+ 43us) client_negotiation.cc:769] Sending connection context
0929 19:03:15.759221 (+ 88us) client_negotiation.cc:240] Negotiation successful
0929 19:03:15.759322 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":86,"threads_started":1}
I20260929 19:03:15.759413 14875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:15.751954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46982 (local address 127.0.98.1:38761)
0929 19:03:15.752078 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:15.752081 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:15.752220 (+ 139us) server_negotiation.cc:407] Connection header received
0929 19:03:15.752301 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:15.752303 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:15.752314 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:15.752389 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:15.752958 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:15.753608 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:15.754559 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:15.754819 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:15.754891 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:15.757001 (+ 2110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:03:15.757008 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:03:15.757008 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:03:15.757014 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:03:15.758257 (+ 1243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:15.758635 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:15.758637 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:15.758637 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:15.758691 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:15.758897 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:15.758898 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:15.758898 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:15.759014 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:03:15.759089 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:15.759254 (+ 165us) server_negotiation.cc:299] Negotiation successful
0929 19:03:15.759327 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":60,"threads_started":1}
I20260929 19:03:15.759845 14801 tablet_service.cc:1514] Processing DeleteTablet for tablet e6f87c19fbaf4c97954b20f41afc0d4f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:03:15 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46982
I20260929 19:03:15.760301 14876 tablet_replica.cc:330] stopping tablet replica
I20260929 19:03:15.760426 14876 raft_consensus.cc:2229] T e6f87c19fbaf4c97954b20f41afc0d4f P 1ec62c6e71024161b5f1f49e6620aa3f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 19:03:15.760504 14876 raft_consensus.cc:2258] T e6f87c19fbaf4c97954b20f41afc0d4f P 1ec62c6e71024161b5f1f49e6620aa3f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 19:03:15.760733 14876 ts_tablet_manager.cc:1905] T e6f87c19fbaf4c97954b20f41afc0d4f P 1ec62c6e71024161b5f1f49e6620aa3f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 19:03:15.762595 14876 ts_tablet_manager.cc:1918] T e6f87c19fbaf4c97954b20f41afc0d4f P 1ec62c6e71024161b5f1f49e6620aa3f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 19:03:15.762696 14876 log.cc:1196] T e6f87c19fbaf4c97954b20f41afc0d4f P 1ec62c6e71024161b5f1f49e6620aa3f: Deleting WAL directory at /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/minicluster-data/ts-0/wal/wals/e6f87c19fbaf4c97954b20f41afc0d4f
I20260929 19:03:15.762913 14876 ts_tablet_manager.cc:1939] T e6f87c19fbaf4c97954b20f41afc0d4f P 1ec62c6e71024161b5f1f49e6620aa3f: Deleting consensus metadata
I20260929 19:03:15.763365 14664 catalog_manager.cc:4841] TS 1ec62c6e71024161b5f1f49e6620aa3f (127.0.98.1:38761): tablet e6f87c19fbaf4c97954b20f41afc0d4f (table test-table [id=dbff969460124ba99f978aa90f296380]) successfully deleted
W20260929 19:03:15.822881 14872 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:03:15.823035 14872 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:03:15.823055 14872 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:03:15.823948 14872 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:03:15.823977 14872 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:03:15.823999 14872 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 19:03:15.825292 14872 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:42897
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=42663
--tserver_master_addrs=127.0.98.62:34379
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14872
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 18:09:13 UTC on ccycloud.quasar-hpkalf.root.comops.site
build id 4680913
I20260929 19:03:15.825479 14872 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:03:15.825631 14872 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:03:15.829985 14884 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:03:15.829984 14881 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:03:15.830044 14872 server_base.cc:1034] running on GCE node
W20260929 19:03:15.830006 14882 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:03:15.830371 14872 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:03:15.830580 14872 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:03:15.831707 14872 hybrid_clock.cc:648] HybridClock initialized: now 1790708595831695 us; error 24 us; skew 500 ppm
Sep 29 19:03:15 dist-test-slave-lqkv krb5kdc[14101](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708595, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:03:15.833856 14872 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 19:03:15.834198 14872 webserver.cc:460] Webserver started at http://127.0.98.2:42663/ using document root <none> and password file <none>
I20260929 19:03:15.834378 14872 fs_manager.cc:359] Metadata directory not provided
I20260929 19:03:15.834429 14872 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:03:15.835513 14872 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 19:03:15.836023 14891 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:15.836179 14872 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 19:03:15.836270 14872 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/minicluster-data/ts-1/wal
uuid: "81a3fe2385c2401cad7e2d8a240a04b0"
format_stamp: "Formatted at 2026-09-29 19:03:15 on dist-test-slave-lqkv"
server_key: "fedbfad63fbe912f1d9b312a62c57d15"
server_key_iv: "4a4e9a95de5686064f3d6f5e4f3a3cb5"
server_key_version: "encryptionkey@0"
I20260929 19:03:15.836580 14872 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:03:15.847656 14872 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:03:15.849243 14872 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:03:15.849377 14872 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:03:15.849566 14872 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:03:15.849814 14872 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:03:15.849871 14872 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:15.849903 14872 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:03:15.849923 14872 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:15.854955 14872 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42897
I20260929 19:03:15.854990 15004 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42897 every 8 connection(s)
I20260929 19:03:15.855270 14872 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/minicluster-data/ts-1/data/info.pb
Sep 29 19:03:15 dist-test-slave-lqkv krb5kdc[14101](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708595, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 19:03:15.860924 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUbnLHE/build/release/bin/kudu as pid 14872
I20260929 19:03:15.861176 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUbnLHE/build/release/bin/kudu
/tmp/dist-test-taskUbnLHE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:46333
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:34379
--webserver_port=37449
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 19:03:15.863157 15007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:15.855946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34379 (local address 127.0.98.2:50835)
0929 19:03:15.856100 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:15.856137 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:15.856535 (+ 398us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:15.856741 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:15.856743 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:15.857272 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:03:15.857430 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:15.857433 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:15.858000 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:15.858000 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:15.858668 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:15.858670 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:15.858959 (+ 289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:15.858959 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:15.859038 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:15.859445 (+ 407us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:03:15.859451 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:03:15.860540 (+ 1089us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:03:15.861952 (+ 1412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:03:15.861958 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:03:15.861960 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:03:15.862222 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:03:15.862446 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:03:15.862447 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:03:15.862447 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:03:15.862527 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:03:15.862773 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:03:15.862776 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:03:15.862859 (+ 83us) client_negotiation.cc:769] Sending connection context
0929 19:03:15.862943 (+ 84us) client_negotiation.cc:240] Negotiation successful
0929 19:03:15.863035 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":80,"threads_started":1}
I20260929 19:03:15.863173 14870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:15.855964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50835 (local address 127.0.98.62:34379)
0929 19:03:15.856018 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:15.856021 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:15.856212 (+ 191us) server_negotiation.cc:407] Connection header received
0929 19:03:15.856620 (+ 408us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:15.856621 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:15.856632 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:15.856690 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:15.857498 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:15.857924 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:15.858730 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:15.858898 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:15.858944 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:15.860614 (+ 1670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:03:15.860615 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:03:15.860616 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:03:15.860621 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:03:15.861882 (+ 1261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:15.862301 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:15.862302 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:15.862302 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:15.862362 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:15.862591 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:15.862591 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:15.862592 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:15.862717 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:03:15.862777 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:15.862984 (+ 207us) server_negotiation.cc:299] Negotiation successful
0929 19:03:15.863076 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260929 19:03:15.863595 15005 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34379
I20260929 19:03:15.863679 15005 heartbeater.cc:464] Registering TS with master...
I20260929 19:03:15.863843 15005 heartbeater.cc:511] Master 127.0.98.62:34379 requested a full tablet report, sending...
I20260929 19:03:15.864162 14679 ts_manager.cc:194] Registered new tserver with Master: 81a3fe2385c2401cad7e2d8a240a04b0 (127.0.98.2:42897)
I20260929 19:03:15.864552 14679 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:50835
W20260929 19:03:15.936749 15008 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:03:15.936946 15008 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:03:15.936968 15008 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:03:15.937989 15008 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:03:15.938020 15008 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:03:15.938043 15008 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 19:03:15.939318 15008 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:46333
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=37449
--tserver_master_addrs=127.0.98.62:34379
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15008
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 18:09:13 UTC on ccycloud.quasar-hpkalf.root.comops.site
build id 4680913
I20260929 19:03:15.939469 15008 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:03:15.939604 15008 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:03:15.944001 15016 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:03:15.944010 15013 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:03:15.944083 15014 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:03:15.944181 15008 server_base.cc:1034] running on GCE node
I20260929 19:03:15.944343 15008 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:03:15.944536 15008 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:03:15.945647 15008 hybrid_clock.cc:648] HybridClock initialized: now 1790708595945614 us; error 43 us; skew 500 ppm
Sep 29 19:03:15 dist-test-slave-lqkv krb5kdc[14101](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708595, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:03:15.947765 15008 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 19:03:15.948076 15008 webserver.cc:460] Webserver started at http://127.0.98.3:37449/ using document root <none> and password file <none>
I20260929 19:03:15.948235 15008 fs_manager.cc:359] Metadata directory not provided
I20260929 19:03:15.948278 15008 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:03:15.949401 15008 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 19:03:15.949856 15023 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:15.949988 15008 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:15.950049 15008 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/minicluster-data/ts-2/wal
uuid: "a83b42620a78478f93bbddcd97803ca4"
format_stamp: "Formatted at 2026-09-29 19:03:15 on dist-test-slave-lqkv"
server_key: "54ccd58312e082582eb3352115ce149b"
server_key_iv: "6b6aec32819450b6002b1434ecdc0e65"
server_key_version: "encryptionkey@0"
I20260929 19:03:15.950276 15008 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:03:15.968168 15008 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:03:15.969795 15008 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:03:15.969884 15008 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:03:15.970026 15008 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:03:15.970222 15008 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:03:15.970248 15008 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:15.970265 15008 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:03:15.970276 15008 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:15.975287 15008 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46333
I20260929 19:03:15.975358 15136 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46333 every 8 connection(s)
I20260929 19:03:15.975654 15008 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/minicluster-data/ts-2/data/info.pb
I20260929 19:03:15.976104 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUbnLHE/build/release/bin/kudu as pid 15008
Sep 29 19:03:15 dist-test-slave-lqkv krb5kdc[14101](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708595, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 19:03:15.982404 15141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:15.976423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51870 (local address 127.0.98.62:34379)
0929 19:03:15.976521 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:15.976523 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:15.976580 (+ 57us) server_negotiation.cc:407] Connection header received
0929 19:03:15.976695 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:15.976696 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:15.976710 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:15.976764 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:15.977291 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:15.977761 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:15.978440 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:15.978639 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:15.978708 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:15.979804 (+ 1096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:03:15.979805 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:03:15.979806 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:03:15.979812 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:03:15.981324 (+ 1512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:15.981646 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:15.981647 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:15.981647 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:15.981694 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:15.981884 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:15.981889 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:15.981889 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:15.982019 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:03:15.982159 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:15.982220 (+ 61us) server_negotiation.cc:299] Negotiation successful
0929 19:03:15.982301 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84,"thread_start_us":47,"threads_started":1}
I20260929 19:03:15.984057 14870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:15.976313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60379 (local address 127.0.98.62:34379)
0929 19:03:15.976356 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:15.976358 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:15.976613 (+ 255us) server_negotiation.cc:407] Connection header received
0929 19:03:15.977097 (+ 484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:15.977098 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:15.977109 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:15.977160 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:15.977927 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:15.978356 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:15.979341 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:15.979525 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:15.979614 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:15.981774 (+ 2160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:03:15.981775 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:03:15.981775 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:03:15.981780 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:03:15.983125 (+ 1345us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:15.983491 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:15.983491 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:15.983492 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:15.983534 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:15.983694 (+ 160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:15.983695 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:15.983695 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:15.983766 (+ 71us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:03:15.983803 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:15.983961 (+ 158us) server_negotiation.cc:299] Negotiation successful
0929 19:03:15.984008 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260929 19:03:15.984241 15139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:15.976329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34379 (local address 127.0.98.3:60379)
0929 19:03:15.976505 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:15.976548 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:15.977003 (+ 455us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:15.977216 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:15.977219 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:15.977679 (+ 460us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:03:15.977826 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:15.977830 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:15.978438 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:15.978439 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:15.979249 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:15.979251 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:15.979686 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:15.979687 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:15.979808 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:15.980220 (+ 412us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:03:15.980227 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:03:15.981697 (+ 1470us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:03:15.983190 (+ 1493us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:03:15.983195 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:03:15.983196 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:03:15.983427 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:03:15.983572 (+ 145us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:03:15.983572 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:03:15.983572 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:03:15.983645 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:03:15.983797 (+ 152us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:03:15.983799 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:03:15.983883 (+ 84us) client_negotiation.cc:769] Sending connection context
0929 19:03:15.983945 (+ 62us) client_negotiation.cc:240] Negotiation successful
0929 19:03:15.984034 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":107,"threads_started":1}
I20260929 19:03:15.984683 15137 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34379
I20260929 19:03:15.984748 15137 heartbeater.cc:464] Registering TS with master...
I20260929 19:03:15.984926 15137 heartbeater.cc:511] Master 127.0.98.62:34379 requested a full tablet report, sending...
I20260929 19:03:15.985231 14679 ts_manager.cc:194] Registered new tserver with Master: a83b42620a78478f93bbddcd97803ca4 (127.0.98.3:46333)
I20260929 19:03:15.985532 14679 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:60379
I20260929 19:03:15.993974 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 19:03:15.996577 14870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:15.994276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51872 (local address 127.0.98.62:34379)
0929 19:03:15.994337 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:15.994339 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:15.994431 (+ 92us) server_negotiation.cc:407] Connection header received
0929 19:03:15.994529 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:15.994530 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:15.994568 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:15.994648 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 19:03:15.994743 (+ 95us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:15.995141 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:15.995845 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:15.996032 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:15.996114 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:15.996284 (+ 170us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 19:03:15.996359 (+ 75us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 19:03:15.996458 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:15.996495 (+ 37us) server_negotiation.cc:299] Negotiation successful
0929 19:03:15.996510 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 19:03:15.996856 14679 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:51872:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 19:03:15.998270 14679 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 19:03:16.000232 14801 tablet_service.cc:1467] Processing CreateTablet for tablet c97c283729124b8697ccdc0849ae101f (DEFAULT_TABLE table=test-table [id=9e0ef91c6de7464a8ea1acf5638b983d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:03:16.000413 14801 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c97c283729124b8697ccdc0849ae101f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:03:16.001902 14759 tablet_bootstrap.cc:492] T c97c283729124b8697ccdc0849ae101f P 1ec62c6e71024161b5f1f49e6620aa3f: Bootstrap starting.
I20260929 19:03:16.002205 14759 tablet_bootstrap.cc:654] T c97c283729124b8697ccdc0849ae101f P 1ec62c6e71024161b5f1f49e6620aa3f: Neither blocks nor log segments found. Creating new log.
I20260929 19:03:16.002966 14759 tablet_bootstrap.cc:492] T c97c283729124b8697ccdc0849ae101f P 1ec62c6e71024161b5f1f49e6620aa3f: No bootstrap required, opened a new log
I20260929 19:03:16.003026 14759 ts_tablet_manager.cc:1397] T c97c283729124b8697ccdc0849ae101f P 1ec62c6e71024161b5f1f49e6620aa3f: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260929 19:03:16.003171 14759 raft_consensus.cc:348] T c97c283729124b8697ccdc0849ae101f P 1ec62c6e71024161b5f1f49e6620aa3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a83b42620a78478f93bbddcd97803ca4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46333 } } peers { permanent_uuid: "81a3fe2385c2401cad7e2d8a240a04b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42897 } } peers { permanent_uuid: "1ec62c6e71024161b5f1f49e6620aa3f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38761 } }
I20260929 19:03:16.003238 14759 raft_consensus.cc:374] T c97c283729124b8697ccdc0849ae101f P 1ec62c6e71024161b5f1f49e6620aa3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:03:16.003288 14759 raft_consensus.cc:729] T c97c283729124b8697ccdc0849ae101f P 1ec62c6e71024161b5f1f49e6620aa3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ec62c6e71024161b5f1f49e6620aa3f, State: Initialized, Role: FOLLOWER
I20260929 19:03:16.003337 14759 consensus_queue.cc:260] T c97c283729124b8697ccdc0849ae101f P 1ec62c6e71024161b5f1f49e6620aa3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a83b42620a78478f93bbddcd97803ca4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46333 } } peers { permanent_uuid: "81a3fe2385c2401cad7e2d8a240a04b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42897 } } peers { permanent_uuid: "1ec62c6e71024161b5f1f49e6620aa3f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38761 } }
I20260929 19:03:16.003422 14759 ts_tablet_manager.cc:1428] T c97c283729124b8697ccdc0849ae101f P 1ec62c6e71024161b5f1f49e6620aa3f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:16.003659 14874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:15.999901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46333 (local address 127.0.0.1:37656)
0929 19:03:15.999932 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:15.999942 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:15.999979 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:16.000683 (+ 704us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:16.000683 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:16.000685 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:03:16.000713 (+ 28us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:16.000714 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:16.001958 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:16.001959 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:16.002648 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:16.002650 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:16.003392 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:16.003393 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:16.003468 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:16.003490 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 19:03:16.003558 (+ 68us) client_negotiation.cc:240] Negotiation successful
0929 19:03:16.003598 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17}
I20260929 19:03:16.003674 15144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:16.000213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37656 (local address 127.0.98.3:46333)
0929 19:03:16.000367 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:16.000371 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:16.000381 (+ 10us) server_negotiation.cc:407] Connection header received
0929 19:03:16.000419 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:16.000422 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:16.000436 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:16.000543 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:03:16.000779 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:16.001839 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:16.002741 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:16.003293 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:16.003383 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:16.003442 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:16.003574 (+ 132us) server_negotiation.cc:299] Negotiation successful
0929 19:03:16.003607 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":82,"threads_started":1}
I20260929 19:03:16.004391 15071 tablet_service.cc:1467] Processing CreateTablet for tablet c97c283729124b8697ccdc0849ae101f (DEFAULT_TABLE table=test-table [id=9e0ef91c6de7464a8ea1acf5638b983d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:03:16.004694 15071 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c97c283729124b8697ccdc0849ae101f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:03:16.004737 15143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:15.999979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42897 (local address 127.0.0.1:44842)
0929 19:03:16.000141 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:16.000150 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:16.000188 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:16.000573 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:16.000574 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:16.000577 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:03:16.000613 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:16.000615 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:16.001644 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:16.001645 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:16.002729 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:16.002733 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:16.004513 (+ 1780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:16.004514 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:16.004578 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:16.004582 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 19:03:16.004638 (+ 56us) client_negotiation.cc:240] Negotiation successful
0929 19:03:16.004674 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":81,"threads_started":1}
I20260929 19:03:16.004760 15145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:16.000202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44842 (local address 127.0.98.2:42897)
0929 19:03:16.000392 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:16.000397 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:16.000407 (+ 10us) server_negotiation.cc:407] Connection header received
0929 19:03:16.000443 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:16.000444 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:16.000480 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:16.000566 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:03:16.000670 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:16.001540 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:16.003641 (+ 2101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:16.004389 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:16.004508 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:16.004564 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:16.004651 (+ 87us) server_negotiation.cc:299] Negotiation successful
0929 19:03:16.004685 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":91,"threads_started":1}
I20260929 19:03:16.005393 14939 tablet_service.cc:1467] Processing CreateTablet for tablet c97c283729124b8697ccdc0849ae101f (DEFAULT_TABLE table=test-table [id=9e0ef91c6de7464a8ea1acf5638b983d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:03:16.005668 14939 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c97c283729124b8697ccdc0849ae101f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:03:16.007527 15147 tablet_bootstrap.cc:492] T c97c283729124b8697ccdc0849ae101f P a83b42620a78478f93bbddcd97803ca4: Bootstrap starting.
I20260929 19:03:16.008185 15147 tablet_bootstrap.cc:654] T c97c283729124b8697ccdc0849ae101f P a83b42620a78478f93bbddcd97803ca4: Neither blocks nor log segments found. Creating new log.
I20260929 19:03:16.008834 15147 log.cc:824] T c97c283729124b8697ccdc0849ae101f P a83b42620a78478f93bbddcd97803ca4: Log is configured to *not* fsync() on all Append() calls
I20260929 19:03:16.008847 15148 tablet_bootstrap.cc:492] T c97c283729124b8697ccdc0849ae101f P 81a3fe2385c2401cad7e2d8a240a04b0: Bootstrap starting.
I20260929 19:03:16.009335 15148 tablet_bootstrap.cc:654] T c97c283729124b8697ccdc0849ae101f P 81a3fe2385c2401cad7e2d8a240a04b0: Neither blocks nor log segments found. Creating new log.
I20260929 19:03:16.009510 15147 tablet_bootstrap.cc:492] T c97c283729124b8697ccdc0849ae101f P a83b42620a78478f93bbddcd97803ca4: No bootstrap required, opened a new log
I20260929 19:03:16.009570 15147 ts_tablet_manager.cc:1397] T c97c283729124b8697ccdc0849ae101f P a83b42620a78478f93bbddcd97803ca4: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 19:03:16.009814 15148 log.cc:824] T c97c283729124b8697ccdc0849ae101f P 81a3fe2385c2401cad7e2d8a240a04b0: Log is configured to *not* fsync() on all Append() calls
I20260929 19:03:16.010476 15148 tablet_bootstrap.cc:492] T c97c283729124b8697ccdc0849ae101f P 81a3fe2385c2401cad7e2d8a240a04b0: No bootstrap required, opened a new log
I20260929 19:03:16.010557 15148 ts_tablet_manager.cc:1397] T c97c283729124b8697ccdc0849ae101f P 81a3fe2385c2401cad7e2d8a240a04b0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 19:03:16.011137 15147 raft_consensus.cc:348] T c97c283729124b8697ccdc0849ae101f P a83b42620a78478f93bbddcd97803ca4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a83b42620a78478f93bbddcd97803ca4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46333 } } peers { permanent_uuid: "81a3fe2385c2401cad7e2d8a240a04b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42897 } } peers { permanent_uuid: "1ec62c6e71024161b5f1f49e6620aa3f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38761 } }
I20260929 19:03:16.011260 15147 raft_consensus.cc:374] T c97c283729124b8697ccdc0849ae101f P a83b42620a78478f93bbddcd97803ca4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:03:16.011281 15147 raft_consensus.cc:729] T c97c283729124b8697ccdc0849ae101f P a83b42620a78478f93bbddcd97803ca4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a83b42620a78478f93bbddcd97803ca4, State: Initialized, Role: FOLLOWER
I20260929 19:03:16.011402 15147 consensus_queue.cc:260] T c97c283729124b8697ccdc0849ae101f P a83b42620a78478f93bbddcd97803ca4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a83b42620a78478f93bbddcd97803ca4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46333 } } peers { permanent_uuid: "81a3fe2385c2401cad7e2d8a240a04b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42897 } } peers { permanent_uuid: "1ec62c6e71024161b5f1f49e6620aa3f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38761 } }
I20260929 19:03:16.011634 15147 ts_tablet_manager.cc:1428] T c97c283729124b8697ccdc0849ae101f P a83b42620a78478f93bbddcd97803ca4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 19:03:16.011678 15137 heartbeater.cc:503] Master 127.0.98.62:34379 was elected leader, sending a full tablet report...
I20260929 19:03:16.012192 15148 raft_consensus.cc:348] T c97c283729124b8697ccdc0849ae101f P 81a3fe2385c2401cad7e2d8a240a04b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a83b42620a78478f93bbddcd97803ca4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46333 } } peers { permanent_uuid: "81a3fe2385c2401cad7e2d8a240a04b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42897 } } peers { permanent_uuid: "1ec62c6e71024161b5f1f49e6620aa3f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38761 } }
I20260929 19:03:16.012337 15148 raft_consensus.cc:374] T c97c283729124b8697ccdc0849ae101f P 81a3fe2385c2401cad7e2d8a240a04b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:03:16.012380 15148 raft_consensus.cc:729] T c97c283729124b8697ccdc0849ae101f P 81a3fe2385c2401cad7e2d8a240a04b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81a3fe2385c2401cad7e2d8a240a04b0, State: Initialized, Role: FOLLOWER
I20260929 19:03:16.012502 15148 consensus_queue.cc:260] T c97c283729124b8697ccdc0849ae101f P 81a3fe2385c2401cad7e2d8a240a04b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a83b42620a78478f93bbddcd97803ca4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46333 } } peers { permanent_uuid: "81a3fe2385c2401cad7e2d8a240a04b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42897 } } peers { permanent_uuid: "1ec62c6e71024161b5f1f49e6620aa3f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38761 } }
I20260929 19:03:16.012769 15148 ts_tablet_manager.cc:1428] T c97c283729124b8697ccdc0849ae101f P 81a3fe2385c2401cad7e2d8a240a04b0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 19:03:16.012769 15005 heartbeater.cc:503] Master 127.0.98.62:34379 was elected leader, sending a full tablet report...
I20260929 19:03:16.032717 15152 raft_consensus.cc:482] T c97c283729124b8697ccdc0849ae101f P 81a3fe2385c2401cad7e2d8a240a04b0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 19:03:16.032801 15152 raft_consensus.cc:504] T c97c283729124b8697ccdc0849ae101f P 81a3fe2385c2401cad7e2d8a240a04b0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a83b42620a78478f93bbddcd97803ca4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46333 } } peers { permanent_uuid: "81a3fe2385c2401cad7e2d8a240a04b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42897 } } peers { permanent_uuid: "1ec62c6e71024161b5f1f49e6620aa3f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38761 } }
I20260929 19:03:16.033143 15152 leader_election.cc:288] T c97c283729124b8697ccdc0849ae101f P 81a3fe2385c2401cad7e2d8a240a04b0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a83b42620a78478f93bbddcd97803ca4 (127.0.98.3:46333), 1ec62c6e71024161b5f1f49e6620aa3f (127.0.98.1:38761)
I20260929 19:03:16.036698 14875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:16.033326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50183 (local address 127.0.98.1:38761)
0929 19:03:16.033368 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:16.033370 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:16.033379 (+ 9us) server_negotiation.cc:407] Connection header received
0929 19:03:16.033460 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:16.033461 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:16.033485 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:16.033561 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:03:16.033700 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:16.034428 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:16.035810 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:16.036383 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:16.036445 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:16.036499 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:16.036592 (+ 93us) server_negotiation.cc:299] Negotiation successful
0929 19:03:16.036637 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260929 19:03:16.036697 15144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:16.033381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59983 (local address 127.0.98.3:46333)
0929 19:03:16.033420 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:16.033422 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:16.033544 (+ 122us) server_negotiation.cc:407] Connection header received
0929 19:03:16.033569 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:16.033570 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:16.033580 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:16.033618 (+ 38us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:03:16.033751 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:16.034406 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:16.035810 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:16.036341 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:16.036419 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:16.036429 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:16.036612 (+ 183us) server_negotiation.cc:299] Negotiation successful
0929 19:03:16.036643 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260929 19:03:16.036737 15007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:16.033288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38761 (local address 127.0.98.2:50183)
0929 19:03:16.033326 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:16.033336 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:16.033372 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:16.033575 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:16.033576 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:16.033578 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:03:16.033633 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:16.033634 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:16.034487 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:16.034488 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:16.035726 (+ 1238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:16.035729 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:16.036453 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:16.036453 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:16.036528 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:16.036530 (+ 2us) client_negotiation.cc:769] Sending connection context
0929 19:03:16.036583 (+ 53us) client_negotiation.cc:240] Negotiation successful
0929 19:03:16.036646 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23,"mutex_wait_us":35}
I20260929 19:03:16.036746 15153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:16.033288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46333 (local address 127.0.98.2:59983)
0929 19:03:16.033449 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:16.033459 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:16.033521 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:16.033630 (+ 109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:16.033630 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:16.033631 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:03:16.033685 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:16.033687 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:16.034487 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:16.034488 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:16.035726 (+ 1238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:16.035729 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:16.036453 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:16.036453 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:16.036526 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:16.036530 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 19:03:16.036583 (+ 53us) client_negotiation.cc:240] Negotiation successful
0929 19:03:16.036645 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"mutex_wait_us":119,"thread_start_us":72,"threads_started":1}
I20260929 19:03:16.037032 14821 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c97c283729124b8697ccdc0849ae101f" candidate_uuid: "81a3fe2385c2401cad7e2d8a240a04b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ec62c6e71024161b5f1f49e6620aa3f" is_pre_election: true
I20260929 19:03:16.037134 15091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c97c283729124b8697ccdc0849ae101f" candidate_uuid: "81a3fe2385c2401cad7e2d8a240a04b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a83b42620a78478f93bbddcd97803ca4" is_pre_election: true
I20260929 19:03:16.037170 14821 raft_consensus.cc:2454] T c97c283729124b8697ccdc0849ae101f P 1ec62c6e71024161b5f1f49e6620aa3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 81a3fe2385c2401cad7e2d8a240a04b0 in term 0.
I20260929 19:03:16.037235 15091 raft_consensus.cc:2454] T c97c283729124b8697ccdc0849ae101f P a83b42620a78478f93bbddcd97803ca4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 81a3fe2385c2401cad7e2d8a240a04b0 in term 0.
I20260929 19:03:16.037418 14894 leader_election.cc:302] T c97c283729124b8697ccdc0849ae101f P 81a3fe2385c2401cad7e2d8a240a04b0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 81a3fe2385c2401cad7e2d8a240a04b0, a83b42620a78478f93bbddcd97803ca4; no voters:
I20260929 19:03:16.037544 15152 raft_consensus.cc:2781] T c97c283729124b8697ccdc0849ae101f P 81a3fe2385c2401cad7e2d8a240a04b0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 19:03:16.037611 15152 raft_consensus.cc:482] T c97c283729124b8697ccdc0849ae101f P 81a3fe2385c2401cad7e2d8a240a04b0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 19:03:16.037627 15152 raft_consensus.cc:3037] T c97c283729124b8697ccdc0849ae101f P 81a3fe2385c2401cad7e2d8a240a04b0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:03:16.038179 15152 raft_consensus.cc:504] T c97c283729124b8697ccdc0849ae101f P 81a3fe2385c2401cad7e2d8a240a04b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a83b42620a78478f93bbddcd97803ca4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46333 } } peers { permanent_uuid: "81a3fe2385c2401cad7e2d8a240a04b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42897 } } peers { permanent_uuid: "1ec62c6e71024161b5f1f49e6620aa3f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38761 } }
I20260929 19:03:16.038303 15152 leader_election.cc:288] T c97c283729124b8697ccdc0849ae101f P 81a3fe2385c2401cad7e2d8a240a04b0 [CANDIDATE]: Term 1 election: Requested vote from peers a83b42620a78478f93bbddcd97803ca4 (127.0.98.3:46333), 1ec62c6e71024161b5f1f49e6620aa3f (127.0.98.1:38761)
I20260929 19:03:16.038442 14821 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c97c283729124b8697ccdc0849ae101f" candidate_uuid: "81a3fe2385c2401cad7e2d8a240a04b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ec62c6e71024161b5f1f49e6620aa3f"
I20260929 19:03:16.038446 15091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c97c283729124b8697ccdc0849ae101f" candidate_uuid: "81a3fe2385c2401cad7e2d8a240a04b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a83b42620a78478f93bbddcd97803ca4"
I20260929 19:03:16.038514 14821 raft_consensus.cc:3037] T c97c283729124b8697ccdc0849ae101f P 1ec62c6e71024161b5f1f49e6620aa3f [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:03:16.038514 15091 raft_consensus.cc:3037] T c97c283729124b8697ccdc0849ae101f P a83b42620a78478f93bbddcd97803ca4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:03:16.039266 14821 raft_consensus.cc:2454] T c97c283729124b8697ccdc0849ae101f P 1ec62c6e71024161b5f1f49e6620aa3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 81a3fe2385c2401cad7e2d8a240a04b0 in term 1.
I20260929 19:03:16.039266 15091 raft_consensus.cc:2454] T c97c283729124b8697ccdc0849ae101f P a83b42620a78478f93bbddcd97803ca4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 81a3fe2385c2401cad7e2d8a240a04b0 in term 1.
I20260929 19:03:16.039446 14893 leader_election.cc:302] T c97c283729124b8697ccdc0849ae101f P 81a3fe2385c2401cad7e2d8a240a04b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ec62c6e71024161b5f1f49e6620aa3f, 81a3fe2385c2401cad7e2d8a240a04b0; no voters:
I20260929 19:03:16.039531 15152 raft_consensus.cc:2781] T c97c283729124b8697ccdc0849ae101f P 81a3fe2385c2401cad7e2d8a240a04b0 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:03:16.039669 15152 raft_consensus.cc:686] T c97c283729124b8697ccdc0849ae101f P 81a3fe2385c2401cad7e2d8a240a04b0 [term 1 LEADER]: Becoming Leader. State: Replica: 81a3fe2385c2401cad7e2d8a240a04b0, State: Running, Role: LEADER
I20260929 19:03:16.039752 15152 consensus_queue.cc:237] T c97c283729124b8697ccdc0849ae101f P 81a3fe2385c2401cad7e2d8a240a04b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a83b42620a78478f93bbddcd97803ca4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46333 } } peers { permanent_uuid: "81a3fe2385c2401cad7e2d8a240a04b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42897 } } peers { permanent_uuid: "1ec62c6e71024161b5f1f49e6620aa3f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38761 } }
I20260929 19:03:16.040421 14678 catalog_manager.cc:5495] T c97c283729124b8697ccdc0849ae101f P 81a3fe2385c2401cad7e2d8a240a04b0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 81a3fe2385c2401cad7e2d8a240a04b0 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "81a3fe2385c2401cad7e2d8a240a04b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a83b42620a78478f93bbddcd97803ca4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46333 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "81a3fe2385c2401cad7e2d8a240a04b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42897 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1ec62c6e71024161b5f1f49e6620aa3f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38761 } health_report { overall_health: UNKNOWN } } }
I20260929 19:03:16.048848 15145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:16.046715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44844 (local address 127.0.98.2:42897)
0929 19:03:16.046753 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:16.046755 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:16.046763 (+ 8us) server_negotiation.cc:407] Connection header received
0929 19:03:16.046848 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:16.046849 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:16.046863 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:16.046923 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 19:03:16.047045 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:16.047427 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:16.048114 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:16.048302 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:16.048381 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:16.048538 (+ 157us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 19:03:16.048618 (+ 80us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 19:03:16.048702 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:16.048730 (+ 28us) server_negotiation.cc:299] Negotiation successful
0929 19:03:16.048746 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260929 19:03:16.050201 15091 raft_consensus.cc:1261] T c97c283729124b8697ccdc0849ae101f P a83b42620a78478f93bbddcd97803ca4 [term 1 FOLLOWER]: Refusing update from remote peer 81a3fe2385c2401cad7e2d8a240a04b0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 19:03:16.050405 14821 raft_consensus.cc:1261] T c97c283729124b8697ccdc0849ae101f P 1ec62c6e71024161b5f1f49e6620aa3f [term 1 FOLLOWER]: Refusing update from remote peer 81a3fe2385c2401cad7e2d8a240a04b0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 19:03:16.050460 15152 consensus_queue.cc:1035] T c97c283729124b8697ccdc0849ae101f P 81a3fe2385c2401cad7e2d8a240a04b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a83b42620a78478f93bbddcd97803ca4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46333 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 19:03:16.050616 15154 consensus_queue.cc:1035] T c97c283729124b8697ccdc0849ae101f P 81a3fe2385c2401cad7e2d8a240a04b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ec62c6e71024161b5f1f49e6620aa3f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 19:03:16.052167 15156 mvcc.cc:205] Tried to move back new op lower bound from 7334742409419485184 to 7334742409379270656. Current Snapshot: MvccSnapshot[applied={T|T < 7334742409419485184}]
I20260929 19:03:16.052675 15158 mvcc.cc:205] Tried to move back new op lower bound from 7334742409419485184 to 7334742409379270656. Current Snapshot: MvccSnapshot[applied={T|T < 7334742409419485184 or (T in {7334742409419485184})}]
I20260929 19:03:16.053160 15157 mvcc.cc:205] Tried to move back new op lower bound from 7334742409419485184 to 7334742409379270656. Current Snapshot: MvccSnapshot[applied={T|T < 7334742409419485184}]
I20260929 19:03:16.054526 14678 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:51872:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 19:03:16.054608 14678 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:51872:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 19:03:16.055255 14678 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0c22890aea31494ea792c6d47109b7e0: Sending DeleteTablet for 3 replicas of tablet c97c283729124b8697ccdc0849ae101f
I20260929 19:03:16.055537 14939 tablet_service.cc:1514] Processing DeleteTablet for tablet c97c283729124b8697ccdc0849ae101f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:03:16 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44842
I20260929 19:03:16.055557 14801 tablet_service.cc:1514] Processing DeleteTablet for tablet c97c283729124b8697ccdc0849ae101f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:03:16 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46982
I20260929 19:03:16.055658 14876 tablet_replica.cc:330] T c97c283729124b8697ccdc0849ae101f P 1ec62c6e71024161b5f1f49e6620aa3f: stopping tablet replica
I20260929 19:03:16.055718 14876 raft_consensus.cc:2229] T c97c283729124b8697ccdc0849ae101f P 1ec62c6e71024161b5f1f49e6620aa3f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 19:03:16.055744 15165 tablet_replica.cc:330] T c97c283729124b8697ccdc0849ae101f P 81a3fe2385c2401cad7e2d8a240a04b0: stopping tablet replica
I20260929 19:03:16.055766 14876 raft_consensus.cc:2258] T c97c283729124b8697ccdc0849ae101f P 1ec62c6e71024161b5f1f49e6620aa3f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 19:03:16.055747 15071 tablet_service.cc:1514] Processing DeleteTablet for tablet c97c283729124b8697ccdc0849ae101f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:03:16 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37656
I20260929 19:03:16.055817 15165 raft_consensus.cc:2229] T c97c283729124b8697ccdc0849ae101f P 81a3fe2385c2401cad7e2d8a240a04b0 [term 1 LEADER]: Raft consensus shutting down.
I20260929 19:03:16.055887 15165 raft_consensus.cc:2258] T c97c283729124b8697ccdc0849ae101f P 81a3fe2385c2401cad7e2d8a240a04b0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 19:03:16.055923 15166 tablet_replica.cc:330] T c97c283729124b8697ccdc0849ae101f P a83b42620a78478f93bbddcd97803ca4: stopping tablet replica
I20260929 19:03:16.055998 15166 raft_consensus.cc:2229] T c97c283729124b8697ccdc0849ae101f P a83b42620a78478f93bbddcd97803ca4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 19:03:16.056077 15166 raft_consensus.cc:2258] T c97c283729124b8697ccdc0849ae101f P a83b42620a78478f93bbddcd97803ca4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 19:03:16.056087 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUbnLHE/build/release/bin/kudu with pid 14718
I20260929 19:03:16.057902 15166 ts_tablet_manager.cc:1905] T c97c283729124b8697ccdc0849ae101f P a83b42620a78478f93bbddcd97803ca4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 19:03:16.059731 15166 ts_tablet_manager.cc:1918] T c97c283729124b8697ccdc0849ae101f P a83b42620a78478f93bbddcd97803ca4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 19:03:16.059836 15166 log.cc:1196] T c97c283729124b8697ccdc0849ae101f P a83b42620a78478f93bbddcd97803ca4: Deleting WAL directory at /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/minicluster-data/ts-2/wal/wals/c97c283729124b8697ccdc0849ae101f
I20260929 19:03:16.060130 15166 ts_tablet_manager.cc:1939] T c97c283729124b8697ccdc0849ae101f P a83b42620a78478f93bbddcd97803ca4: Deleting consensus metadata
I20260929 19:03:16.060534 14665 catalog_manager.cc:4841] TS a83b42620a78478f93bbddcd97803ca4 (127.0.98.3:46333): tablet c97c283729124b8697ccdc0849ae101f (table test-table [id=9e0ef91c6de7464a8ea1acf5638b983d]) successfully deleted
W20260929 19:03:16.062209 14664 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:38761 (error 108)
I20260929 19:03:16.062577 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUbnLHE/build/release/bin/kudu with pid 14872
I20260929 19:03:16.062639 15143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:16.062543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38761 (local address 127.0.0.1:46986)
0929 19:03:16.062580 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:16.062610 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38761: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
W20260929 19:03:16.063162 14664 catalog_manager.cc:4584] TS 1ec62c6e71024161b5f1f49e6620aa3f (127.0.98.1:38761): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c97c283729124b8697ccdc0849ae101f: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38761: connect: Connection refused (error 111)
I20260929 19:03:16.066946 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUbnLHE/build/release/bin/kudu with pid 15008
I20260929 19:03:16.066960 15143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:16.066857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42897 (local address 127.0.0.1:44852)
0929 19:03:16.066908 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:16.066942 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:42897: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260929 19:03:16.071892 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUbnLHE/build/release/bin/kudu with pid 14646
I20260929 19:03:16.076761 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUbnLHE/build/release/bin/kudu
/tmp/dist-test-taskUbnLHE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34379
--webserver_interface=127.0.98.62
--webserver_port=33305
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34379
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:03:16.153231 15168 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:03:16.153422 15168 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:03:16.153443 15168 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:03:16.154588 15168 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 19:03:16.154630 15168 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:03:16.154644 15168 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:03:16.154654 15168 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 19:03:16.154665 15168 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 19:03:16.156070 15168 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34379
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34379
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=33305
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15168
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 18:09:13 UTC on ccycloud.quasar-hpkalf.root.comops.site
build id 4680913
I20260929 19:03:16.156314 15168 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:03:16.156536 15168 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:03:16.161185 15173 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:03:16.161283 15168 server_base.cc:1034] running on GCE node
W20260929 19:03:16.161182 15174 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:03:16.161183 15176 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:03:16.161580 15168 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:03:16.161850 15168 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:03:16.163007 15168 hybrid_clock.cc:648] HybridClock initialized: now 1790708596162995 us; error 23 us; skew 500 ppm
Sep 29 19:03:16 dist-test-slave-lqkv krb5kdc[14101](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708596, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:03:16.165318 15168 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 19:03:16.165742 15168 webserver.cc:460] Webserver started at http://127.0.98.62:33305/ using document root <none> and password file <none>
I20260929 19:03:16.165972 15168 fs_manager.cc:359] Metadata directory not provided
I20260929 19:03:16.166038 15168 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:03:16.167209 15168 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 19:03:16.167819 15183 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:16.168007 15168 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 19:03:16.168107 15168 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/minicluster-data/master-0/wal
uuid: "0c22890aea31494ea792c6d47109b7e0"
format_stamp: "Formatted at 2026-09-29 19:03:14 on dist-test-slave-lqkv"
server_key: "2fc699f4d0b98c0190e7fb5a1d64f91c"
server_key_iv: "3b7ccfe52eebb0b6bf119b828e6880a7"
server_key_version: "encryptionkey@0"
I20260929 19:03:16.168401 15168 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:03:16.175657 15168 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:03:16.177325 15168 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:03:16.177464 15168 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:03:16.181205 15168 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34379
I20260929 19:03:16.181231 15235 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34379 every 8 connection(s)
I20260929 19:03:16.181541 15168 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/minicluster-data/master-0/data/info.pb
I20260929 19:03:16.182773 15236 sys_catalog.cc:263] Verifying existing consensus state
I20260929 19:03:16.183316 15236 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c22890aea31494ea792c6d47109b7e0: Bootstrap starting.
I20260929 19:03:16.191052 15236 log.cc:824] T 00000000000000000000000000000000 P 0c22890aea31494ea792c6d47109b7e0: Log is configured to *not* fsync() on all Append() calls
I20260929 19:03:16.191233 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUbnLHE/build/release/bin/kudu as pid 15168
I20260929 19:03:16.191473 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUbnLHE/build/release/bin/kudu
/tmp/dist-test-taskUbnLHE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:38761
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:34379
--webserver_port=37081
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 19:03:16.193682 15236 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c22890aea31494ea792c6d47109b7e0: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 19:03:16.193949 15236 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c22890aea31494ea792c6d47109b7e0: Bootstrap complete.
I20260929 19:03:16.195990 15236 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0c22890aea31494ea792c6d47109b7e0 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c22890aea31494ea792c6d47109b7e0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34379 } }
I20260929 19:03:16.196185 15236 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0c22890aea31494ea792c6d47109b7e0 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c22890aea31494ea792c6d47109b7e0, State: Initialized, Role: FOLLOWER
I20260929 19:03:16.196339 15236 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0c22890aea31494ea792c6d47109b7e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c22890aea31494ea792c6d47109b7e0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34379 } }
I20260929 19:03:16.196395 15236 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0c22890aea31494ea792c6d47109b7e0 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 19:03:16.196457 15236 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0c22890aea31494ea792c6d47109b7e0 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 19:03:16.196499 15236 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0c22890aea31494ea792c6d47109b7e0 [term 2 FOLLOWER]: Advancing to term 3
I20260929 19:03:16.197499 15236 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0c22890aea31494ea792c6d47109b7e0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c22890aea31494ea792c6d47109b7e0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34379 } }
I20260929 19:03:16.197577 15236 leader_election.cc:302] T 00000000000000000000000000000000 P 0c22890aea31494ea792c6d47109b7e0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0c22890aea31494ea792c6d47109b7e0; no voters:
I20260929 19:03:16.197779 15236 leader_election.cc:288] T 00000000000000000000000000000000 P 0c22890aea31494ea792c6d47109b7e0 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260929 19:03:16.197809 15240 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0c22890aea31494ea792c6d47109b7e0 [term 3 FOLLOWER]: Leader election won for term 3
I20260929 19:03:16.197918 15240 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0c22890aea31494ea792c6d47109b7e0 [term 3 LEADER]: Becoming Leader. State: Replica: 0c22890aea31494ea792c6d47109b7e0, State: Running, Role: LEADER
I20260929 19:03:16.197996 15240 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0c22890aea31494ea792c6d47109b7e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c22890aea31494ea792c6d47109b7e0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34379 } }
I20260929 19:03:16.198022 15236 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0c22890aea31494ea792c6d47109b7e0 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 19:03:16.198892 15242 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c22890aea31494ea792c6d47109b7e0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "0c22890aea31494ea792c6d47109b7e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c22890aea31494ea792c6d47109b7e0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34379 } } }
I20260929 19:03:16.198899 15241 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c22890aea31494ea792c6d47109b7e0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0c22890aea31494ea792c6d47109b7e0. Latest consensus state: current_term: 3 leader_uuid: "0c22890aea31494ea792c6d47109b7e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c22890aea31494ea792c6d47109b7e0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34379 } } }
I20260929 19:03:16.199005 15241 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c22890aea31494ea792c6d47109b7e0 [sys.catalog]: This master's current role is: LEADER
I20260929 19:03:16.199146 15242 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c22890aea31494ea792c6d47109b7e0 [sys.catalog]: This master's current role is: LEADER
I20260929 19:03:16.199222 15252 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 19:03:16.200021 15252 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 19:03:16.200125 15252 catalog_manager.cc:1259] Loaded cluster ID: 1267f941b71f420d9709dc2a7f743fb7
I20260929 19:03:16.200160 15252 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260929 19:03:16.200350 15252 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f978abd99ad google::LogMessage::Fail() at ??:0
@ 0x7f978abdc88a google::LogMessage::SendToLog() at ??:0
@ 0x7f978abd94de google::LogMessage::Flush() at ??:0
@ 0x7f978abdaa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f9792afa0f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f9792b038b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f978caa70d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f978ca9c954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f978d9006db start_thread at ??:0
@ 0x7f97896d761f clone at ??:0
W20260929 19:03:16.280867 15239 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:03:16.281037 15239 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:03:16.281059 15239 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:03:16.282064 15239 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:03:16.282099 15239 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:03:16.282121 15239 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 19:03:16.283412 15239 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:38761
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=37081
--tserver_master_addrs=127.0.98.62:34379
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15239
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 18:09:13 UTC on ccycloud.quasar-hpkalf.root.comops.site
build id 4680913
I20260929 19:03:16.283625 15239 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:03:16.283842 15239 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:03:16.289254 15272 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:03:16.289403 15273 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:03:16.289286 15275 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:03:16.290194 15239 server_base.cc:1034] running on GCE node
I20260929 19:03:16.290374 15239 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:03:16.290659 15239 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:03:16.291800 15239 hybrid_clock.cc:648] HybridClock initialized: now 1790708596291790 us; error 27 us; skew 500 ppm
Sep 29 19:03:16 dist-test-slave-lqkv krb5kdc[14101](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708596, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:03:16.294353 15239 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 19:03:16.294750 15239 webserver.cc:460] Webserver started at http://127.0.98.1:37081/ using document root <none> and password file <none>
I20260929 19:03:16.294945 15239 fs_manager.cc:359] Metadata directory not provided
I20260929 19:03:16.295006 15239 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:03:16.296241 15239 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 19:03:16.297003 15282 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:16.297155 15239 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 19:03:16.297251 15239 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/minicluster-data/ts-0/wal
uuid: "1ec62c6e71024161b5f1f49e6620aa3f"
format_stamp: "Formatted at 2026-09-29 19:03:14 on dist-test-slave-lqkv"
server_key: "93f2f8110ad1d37608b91bf0c68cfad4"
server_key_iv: "67dcdde50b640040b7694967eaa69b39"
server_key_version: "encryptionkey@0"
I20260929 19:03:16.297549 15239 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:03:16.309715 15239 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:03:16.311672 15239 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:03:16.311796 15239 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:03:16.312023 15239 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:03:16.312410 15289 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260929 19:03:16.314018 15239 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260929 19:03:16.314071 15239 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260929 19:03:16.314109 15239 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260929 19:03:16.314615 15239 ts_tablet_manager.cc:610] Registered 1 tablets
I20260929 19:03:16.314663 15239 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260929 19:03:16.314742 15289 tablet_bootstrap.cc:492] T c97c283729124b8697ccdc0849ae101f P 1ec62c6e71024161b5f1f49e6620aa3f: Bootstrap starting.
I20260929 19:03:16.321504 15239 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38761
I20260929 19:03:16.321823 15239 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/minicluster-data/ts-0/data/info.pb
I20260929 19:03:16.324985 15396 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38761 every 8 connection(s)
I20260929 19:03:16.325191 15289 log.cc:824] T c97c283729124b8697ccdc0849ae101f P 1ec62c6e71024161b5f1f49e6620aa3f: Log is configured to *not* fsync() on all Append() calls
I20260929 19:03:16.326444 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUbnLHE/build/release/bin/kudu as pid 15239
I20260929 19:03:16.326735 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUbnLHE/build/release/bin/kudu
/tmp/dist-test-taskUbnLHE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:42897
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:34379
--webserver_port=42663
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 19:03:16.327592 15289 tablet_bootstrap.cc:492] T c97c283729124b8697ccdc0849ae101f P 1ec62c6e71024161b5f1f49e6620aa3f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 19:03:16.327872 15289 tablet_bootstrap.cc:492] T c97c283729124b8697ccdc0849ae101f P 1ec62c6e71024161b5f1f49e6620aa3f: Bootstrap complete.
I20260929 19:03:16.329070 15289 ts_tablet_manager.cc:1397] T c97c283729124b8697ccdc0849ae101f P 1ec62c6e71024161b5f1f49e6620aa3f: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.011s
I20260929 19:03:16.330283 15289 raft_consensus.cc:348] T c97c283729124b8697ccdc0849ae101f P 1ec62c6e71024161b5f1f49e6620aa3f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a83b42620a78478f93bbddcd97803ca4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46333 } } peers { permanent_uuid: "81a3fe2385c2401cad7e2d8a240a04b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42897 } } peers { permanent_uuid: "1ec62c6e71024161b5f1f49e6620aa3f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38761 } }
I20260929 19:03:16.330510 15289 raft_consensus.cc:729] T c97c283729124b8697ccdc0849ae101f P 1ec62c6e71024161b5f1f49e6620aa3f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ec62c6e71024161b5f1f49e6620aa3f, State: Initialized, Role: FOLLOWER
I20260929 19:03:16.330629 15289 consensus_queue.cc:260] T c97c283729124b8697ccdc0849ae101f P 1ec62c6e71024161b5f1f49e6620aa3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a83b42620a78478f93bbddcd97803ca4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46333 } } peers { permanent_uuid: "81a3fe2385c2401cad7e2d8a240a04b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42897 } } peers { permanent_uuid: "1ec62c6e71024161b5f1f49e6620aa3f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38761 } }
I20260929 19:03:16.330873 15289 ts_tablet_manager.cc:1428] T c97c283729124b8697ccdc0849ae101f P 1ec62c6e71024161b5f1f49e6620aa3f: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
W20260929 19:03:16.382719 15399 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:03:16.322403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34379 (local address 127.0.98.1:47717)
0929 19:03:16.322591 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:16.322628 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:16.323071 (+ 443us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:16.382537 (+ 59466us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34379: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":127,"threads_started":1}
W20260929 19:03:16.382964 15397 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:34379 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:34379: Client connection negotiation failed: client connection to 127.0.98.62:34379: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 19:03:16.383265 15399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:16.383157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34379 (local address 127.0.98.1:48673)
0929 19:03:16.383208 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:16.383244 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260929 19:03:16.383585 15399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:16.383503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34379 (local address 127.0.98.1:60177)
0929 19:03:16.383544 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:16.383568 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260929 19:03:16.383688 15397 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 19:03:16.403573 15401 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:03:16.403749 15401 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:03:16.403769 15401 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:03:16.404939 15401 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:03:16.404982 15401 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:03:16.405028 15401 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 19:03:16.406433 15401 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:42897
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=42663
--tserver_master_addrs=127.0.98.62:34379
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15401
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 18:09:13 UTC on ccycloud.quasar-hpkalf.root.comops.site
build id 4680913
I20260929 19:03:16.406662 15401 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:03:16.406858 15401 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:03:16.412434 15407 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:03:16.412340 15410 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:03:16.412411 15401 server_base.cc:1034] running on GCE node
W20260929 19:03:16.412571 15408 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:03:16.412772 15401 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:03:16.413079 15401 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:03:16.414230 15401 hybrid_clock.cc:648] HybridClock initialized: now 1790708596414215 us; error 26 us; skew 500 ppm
Sep 29 19:03:16 dist-test-slave-lqkv krb5kdc[14101](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708596, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:03:16.416539 15401 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 19:03:16.417024 15401 webserver.cc:460] Webserver started at http://127.0.98.2:42663/ using document root <none> and password file <none>
I20260929 19:03:16.417201 15401 fs_manager.cc:359] Metadata directory not provided
I20260929 19:03:16.417238 15401 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:03:16.418342 15401 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 19:03:16.418910 15417 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:16.419075 15401 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 19:03:16.419153 15401 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/minicluster-data/ts-1/wal
uuid: "81a3fe2385c2401cad7e2d8a240a04b0"
format_stamp: "Formatted at 2026-09-29 19:03:15 on dist-test-slave-lqkv"
server_key: "fedbfad63fbe912f1d9b312a62c57d15"
server_key_iv: "4a4e9a95de5686064f3d6f5e4f3a3cb5"
server_key_version: "encryptionkey@0"
I20260929 19:03:16.419409 15401 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:03:16.439453 15401 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:03:16.441042 15401 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:03:16.441135 15401 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:03:16.441336 15401 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:03:16.441695 15424 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260929 19:03:16.442891 15401 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260929 19:03:16.442932 15401 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.001s user 0.000s sys 0.000s
I20260929 19:03:16.442953 15401 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260929 19:03:16.443506 15401 ts_tablet_manager.cc:610] Registered 1 tablets
I20260929 19:03:16.443535 15401 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260929 19:03:16.443678 15424 tablet_bootstrap.cc:492] T c97c283729124b8697ccdc0849ae101f P 81a3fe2385c2401cad7e2d8a240a04b0: Bootstrap starting.
I20260929 19:03:16.449801 15401 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42897
I20260929 19:03:16.449882 15531 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42897 every 8 connection(s)
I20260929 19:03:16.450139 15401 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/minicluster-data/ts-1/data/info.pb
I20260929 19:03:16.451143 15534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:16.450788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34379 (local address 127.0.98.2:43389)
0929 19:03:16.450973 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:16.451031 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":109,"threads_started":1}
W20260929 19:03:16.451345 15532 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:34379 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:34379: Client connection negotiation failed: client connection to 127.0.98.62:34379: connect: Connection refused (error 111)
I20260929 19:03:16.451589 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUbnLHE/build/release/bin/kudu as pid 15401
I20260929 19:03:16.451654 15534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:16.451519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34379 (local address 127.0.98.2:37941)
0929 19:03:16.451576 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:16.451620 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260929 19:03:16.451895 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUbnLHE/build/release/bin/kudu
/tmp/dist-test-taskUbnLHE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:46333
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:34379
--webserver_port=37449
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 19:03:16.451953 15534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:16.451852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34379 (local address 127.0.98.2:53183)
0929 19:03:16.451898 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:16.451925 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260929 19:03:16.452036 15532 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 19:03:16.453995 15424 log.cc:824] T c97c283729124b8697ccdc0849ae101f P 81a3fe2385c2401cad7e2d8a240a04b0: Log is configured to *not* fsync() on all Append() calls
I20260929 19:03:16.455807 15424 tablet_bootstrap.cc:492] T c97c283729124b8697ccdc0849ae101f P 81a3fe2385c2401cad7e2d8a240a04b0: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 19:03:16.456076 15424 tablet_bootstrap.cc:492] T c97c283729124b8697ccdc0849ae101f P 81a3fe2385c2401cad7e2d8a240a04b0: Bootstrap complete.
I20260929 19:03:16.457012 15424 ts_tablet_manager.cc:1397] T c97c283729124b8697ccdc0849ae101f P 81a3fe2385c2401cad7e2d8a240a04b0: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.006s
I20260929 19:03:16.458034 15424 raft_consensus.cc:348] T c97c283729124b8697ccdc0849ae101f P 81a3fe2385c2401cad7e2d8a240a04b0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a83b42620a78478f93bbddcd97803ca4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46333 } } peers { permanent_uuid: "81a3fe2385c2401cad7e2d8a240a04b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42897 } } peers { permanent_uuid: "1ec62c6e71024161b5f1f49e6620aa3f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38761 } }
I20260929 19:03:16.458248 15424 raft_consensus.cc:729] T c97c283729124b8697ccdc0849ae101f P 81a3fe2385c2401cad7e2d8a240a04b0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81a3fe2385c2401cad7e2d8a240a04b0, State: Initialized, Role: FOLLOWER
I20260929 19:03:16.458381 15424 consensus_queue.cc:260] T c97c283729124b8697ccdc0849ae101f P 81a3fe2385c2401cad7e2d8a240a04b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a83b42620a78478f93bbddcd97803ca4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46333 } } peers { permanent_uuid: "81a3fe2385c2401cad7e2d8a240a04b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42897 } } peers { permanent_uuid: "1ec62c6e71024161b5f1f49e6620aa3f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38761 } }
I20260929 19:03:16.458627 15424 ts_tablet_manager.cc:1428] T c97c283729124b8697ccdc0849ae101f P 81a3fe2385c2401cad7e2d8a240a04b0: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260929 19:03:16.458854 15534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:16.458746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34379 (local address 127.0.98.2:45079)
0929 19:03:16.458789 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:16.458835 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260929 19:03:16.459134 15534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:16.459058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34379 (local address 127.0.98.2:45195)
0929 19:03:16.459097 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:16.459117 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260929 19:03:16.526073 15535 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:03:16.526273 15535 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:03:16.526314 15535 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:03:16.527381 15535 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:03:16.527439 15535 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:03:16.527477 15535 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 19:03:16.528744 15535 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:46333
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=37449
--tserver_master_addrs=127.0.98.62:34379
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15535
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 18:09:13 UTC on ccycloud.quasar-hpkalf.root.comops.site
build id 4680913
I20260929 19:03:16.528985 15535 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:03:16.529178 15535 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:03:16.533444 15542 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:03:16.533450 15543 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:03:16.533504 15545 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:03:16.533653 15535 server_base.cc:1034] running on GCE node
I20260929 19:03:16.533852 15535 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:03:16.534080 15535 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:03:16.535226 15535 hybrid_clock.cc:648] HybridClock initialized: now 1790708596535204 us; error 36 us; skew 500 ppm
Sep 29 19:03:16 dist-test-slave-lqkv krb5kdc[14101](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708596, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:03:16.537521 15535 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 19:03:16.537892 15535 webserver.cc:460] Webserver started at http://127.0.98.3:37449/ using document root <none> and password file <none>
I20260929 19:03:16.538112 15535 fs_manager.cc:359] Metadata directory not provided
I20260929 19:03:16.538172 15535 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:03:16.539336 15535 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260929 19:03:16.539809 15552 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:16.539948 15535 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:16.540031 15535 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/minicluster-data/ts-2/wal
uuid: "a83b42620a78478f93bbddcd97803ca4"
format_stamp: "Formatted at 2026-09-29 19:03:15 on dist-test-slave-lqkv"
server_key: "54ccd58312e082582eb3352115ce149b"
server_key_iv: "6b6aec32819450b6002b1434ecdc0e65"
server_key_version: "encryptionkey@0"
I20260929 19:03:16.540297 15535 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:03:16.550704 15535 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:03:16.552220 15535 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:03:16.552362 15535 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:03:16.552549 15535 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:03:16.552866 15535 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:03:16.552925 15535 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:16.552973 15535 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:03:16.552999 15535 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:16.558425 15535 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46333
I20260929 19:03:16.558480 15665 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46333 every 8 connection(s)
I20260929 19:03:16.558738 15535 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790708447806542-392-0/minicluster-data/ts-2/data/info.pb
I20260929 19:03:16.559553 15668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:16.559249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34379 (local address 127.0.98.3:40917)
0929 19:03:16.559383 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:16.559443 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":63,"threads_started":1}
W20260929 19:03:16.559878 15666 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:34379 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:34379: Client connection negotiation failed: client connection to 127.0.98.62:34379: connect: Connection refused (error 111)
I20260929 19:03:16.560110 15668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:16.560033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34379 (local address 127.0.98.3:54589)
0929 19:03:16.560073 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:16.560095 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260929 19:03:16.560354 15668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:16.560288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34379 (local address 127.0.98.3:54343)
0929 19:03:16.560323 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:16.560342 (+ 19us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260929 19:03:16.560456 15666 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 19:03:16.566561 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUbnLHE/build/release/bin/kudu as pid 15535
I20260929 19:03:17.384177 15670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:17.383974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34379 (local address 127.0.98.1:53223)
0929 19:03:17.384092 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:17.384124 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":61,"threads_started":1}
I20260929 19:03:17.459702 15671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:17.459430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34379 (local address 127.0.98.2:48337)
0929 19:03:17.459633 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:17.459663 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":48,"threads_started":1}
I20260929 19:03:17.560935 15672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:17.560719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34379 (local address 127.0.98.3:53103)
0929 19:03:17.560866 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:17.560895 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":49,"threads_started":1}
I20260929 19:03:17.894419 15673 raft_consensus.cc:482] T c97c283729124b8697ccdc0849ae101f P 1ec62c6e71024161b5f1f49e6620aa3f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 19:03:17.894522 15673 raft_consensus.cc:504] T c97c283729124b8697ccdc0849ae101f P 1ec62c6e71024161b5f1f49e6620aa3f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a83b42620a78478f93bbddcd97803ca4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46333 } } peers { permanent_uuid: "81a3fe2385c2401cad7e2d8a240a04b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42897 } } peers { permanent_uuid: "1ec62c6e71024161b5f1f49e6620aa3f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38761 } }
I20260929 19:03:17.894778 15673 leader_election.cc:288] T c97c283729124b8697ccdc0849ae101f P 1ec62c6e71024161b5f1f49e6620aa3f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a83b42620a78478f93bbddcd97803ca4 (127.0.98.3:46333), 81a3fe2385c2401cad7e2d8a240a04b0 (127.0.98.2:42897)
Sep 29 19:03:17 dist-test-slave-lqkv krb5kdc[14101](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708596, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
Sep 29 19:03:17 dist-test-slave-lqkv krb5kdc[14101](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708596, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
I20260929 19:03:17.904337 15675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:17.894910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46333 (local address 127.0.98.1:50365)
0929 19:03:17.895049 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:17.895061 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:17.895121 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:17.895991 (+ 870us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:17.895991 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:17.896165 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:03:17.896216 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:17.896222 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:17.897319 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:17.897320 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:17.898244 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:17.898246 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:17.898792 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:17.898793 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:17.898901 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:17.899488 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:03:17.899493 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:03:17.901301 (+ 1808us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:03:17.902931 (+ 1630us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:03:17.902931 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:03:17.902932 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:03:17.903150 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:03:17.903383 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:03:17.903384 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:03:17.903385 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:03:17.903502 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:03:17.903861 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:03:17.903868 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:03:17.903994 (+ 126us) client_negotiation.cc:769] Sending connection context
0929 19:03:17.904117 (+ 123us) client_negotiation.cc:240] Negotiation successful
0929 19:03:17.904241 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"mutex_wait_us":14,"thread_start_us":32,"threads_started":1}
I20260929 19:03:17.904367 15677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:17.895104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34837 (local address 127.0.98.2:42897)
0929 19:03:17.895239 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:17.895247 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:17.895264 (+ 17us) server_negotiation.cc:407] Connection header received
0929 19:03:17.895361 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:17.895363 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:17.895401 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:17.895494 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:17.896322 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:17.897254 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:17.898318 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:17.898700 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:17.898821 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:17.900996 (+ 2175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:03:17.901000 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:03:17.901000 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:03:17.901010 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:03:17.902755 (+ 1745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:17.903250 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:17.903253 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:17.903253 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:17.903341 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:17.903565 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:17.903566 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:17.903567 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:17.903793 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:03:17.903879 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:17.904146 (+ 267us) server_negotiation.cc:299] Negotiation successful
0929 19:03:17.904262 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":48,"threads_started":1}
I20260929 19:03:17.904466 15676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:17.895029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50365 (local address 127.0.98.3:46333)
0929 19:03:17.895201 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:17.895214 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:17.895259 (+ 45us) server_negotiation.cc:407] Connection header received
0929 19:03:17.895821 (+ 562us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:17.895824 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:17.895871 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:17.896003 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:17.896323 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:17.897231 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:17.898338 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:17.898702 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:17.898821 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:17.901393 (+ 2572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:03:17.901396 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:03:17.901396 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:03:17.901404 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:03:17.902855 (+ 1451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:17.903221 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:17.903225 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:17.903225 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:17.903320 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:17.903588 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:17.903589 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:17.903590 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:17.903790 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:03:17.903879 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:17.904183 (+ 304us) server_negotiation.cc:299] Negotiation successful
0929 19:03:17.904319 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":61,"threads_started":1}
I20260929 19:03:17.904557 15674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:17.894910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42897 (local address 127.0.98.1:34837)
0929 19:03:17.895013 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:17.895023 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:17.895078 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:17.895558 (+ 480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:17.895561 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:17.895959 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:03:17.896215 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:17.896222 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:17.897366 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:17.897367 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:17.898242 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:17.898244 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:17.898798 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:17.898799 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:17.898902 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:17.899440 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:03:17.899447 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:03:17.900899 (+ 1452us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:03:17.902838 (+ 1939us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:03:17.902844 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:03:17.902846 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:03:17.903152 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:03:17.903435 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:03:17.903436 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:03:17.903436 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:03:17.903502 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:03:17.903863 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:03:17.903869 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:03:17.903997 (+ 128us) client_negotiation.cc:769] Sending connection context
0929 19:03:17.904073 (+ 76us) client_negotiation.cc:240] Negotiation successful
0929 19:03:17.904322 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90,"mutex_wait_us":159,"thread_start_us":44,"threads_started":1}
I20260929 19:03:17.904980 15486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c97c283729124b8697ccdc0849ae101f" candidate_uuid: "1ec62c6e71024161b5f1f49e6620aa3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "81a3fe2385c2401cad7e2d8a240a04b0" is_pre_election: true
I20260929 19:03:17.905130 15486 raft_consensus.cc:2454] T c97c283729124b8697ccdc0849ae101f P 81a3fe2385c2401cad7e2d8a240a04b0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ec62c6e71024161b5f1f49e6620aa3f in term 1.
I20260929 19:03:17.905411 15286 leader_election.cc:302] T c97c283729124b8697ccdc0849ae101f P 1ec62c6e71024161b5f1f49e6620aa3f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ec62c6e71024161b5f1f49e6620aa3f, 81a3fe2385c2401cad7e2d8a240a04b0; no voters:
W20260929 19:03:17.905480 15285 leader_election.cc:341] T c97c283729124b8697ccdc0849ae101f P 1ec62c6e71024161b5f1f49e6620aa3f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a83b42620a78478f93bbddcd97803ca4 (127.0.98.3:46333): Not found: Tablet not found: c97c283729124b8697ccdc0849ae101f
I20260929 19:03:17.905522 15673 raft_consensus.cc:2781] T c97c283729124b8697ccdc0849ae101f P 1ec62c6e71024161b5f1f49e6620aa3f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260929 19:03:17.905596 15673 raft_consensus.cc:482] T c97c283729124b8697ccdc0849ae101f P 1ec62c6e71024161b5f1f49e6620aa3f [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 19:03:17.905628 15673 raft_consensus.cc:3037] T c97c283729124b8697ccdc0849ae101f P 1ec62c6e71024161b5f1f49e6620aa3f [term 1 FOLLOWER]: Advancing to term 2
I20260929 19:03:17.906661 15673 raft_consensus.cc:504] T c97c283729124b8697ccdc0849ae101f P 1ec62c6e71024161b5f1f49e6620aa3f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a83b42620a78478f93bbddcd97803ca4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46333 } } peers { permanent_uuid: "81a3fe2385c2401cad7e2d8a240a04b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42897 } } peers { permanent_uuid: "1ec62c6e71024161b5f1f49e6620aa3f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38761 } }
I20260929 19:03:17.906798 15673 leader_election.cc:288] T c97c283729124b8697ccdc0849ae101f P 1ec62c6e71024161b5f1f49e6620aa3f [CANDIDATE]: Term 2 election: Requested vote from peers a83b42620a78478f93bbddcd97803ca4 (127.0.98.3:46333), 81a3fe2385c2401cad7e2d8a240a04b0 (127.0.98.2:42897)
I20260929 19:03:17.907012 15486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c97c283729124b8697ccdc0849ae101f" candidate_uuid: "1ec62c6e71024161b5f1f49e6620aa3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "81a3fe2385c2401cad7e2d8a240a04b0"
W20260929 19:03:17.907073 15285 leader_election.cc:341] T c97c283729124b8697ccdc0849ae101f P 1ec62c6e71024161b5f1f49e6620aa3f [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer a83b42620a78478f93bbddcd97803ca4 (127.0.98.3:46333): Not found: Tablet not found: c97c283729124b8697ccdc0849ae101f
I20260929 19:03:17.907099 15486 raft_consensus.cc:3037] T c97c283729124b8697ccdc0849ae101f P 81a3fe2385c2401cad7e2d8a240a04b0 [term 1 FOLLOWER]: Advancing to term 2
I20260929 19:03:17.908088 15486 raft_consensus.cc:2454] T c97c283729124b8697ccdc0849ae101f P 81a3fe2385c2401cad7e2d8a240a04b0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ec62c6e71024161b5f1f49e6620aa3f in term 2.
I20260929 19:03:17.908265 15286 leader_election.cc:302] T c97c283729124b8697ccdc0849ae101f P 1ec62c6e71024161b5f1f49e6620aa3f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1ec62c6e71024161b5f1f49e6620aa3f, 81a3fe2385c2401cad7e2d8a240a04b0; no voters: a83b42620a78478f93bbddcd97803ca4
I20260929 19:03:17.908412 15673 raft_consensus.cc:2781] T c97c283729124b8697ccdc0849ae101f P 1ec62c6e71024161b5f1f49e6620aa3f [term 2 FOLLOWER]: Leader election won for term 2
I20260929 19:03:17.908525 15673 raft_consensus.cc:686] T c97c283729124b8697ccdc0849ae101f P 1ec62c6e71024161b5f1f49e6620aa3f [term 2 LEADER]: Becoming Leader. State: Replica: 1ec62c6e71024161b5f1f49e6620aa3f, State: Running, Role: LEADER
I20260929 19:03:17.908620 15673 consensus_queue.cc:237] T c97c283729124b8697ccdc0849ae101f P 1ec62c6e71024161b5f1f49e6620aa3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a83b42620a78478f93bbddcd97803ca4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46333 } } peers { permanent_uuid: "81a3fe2385c2401cad7e2d8a240a04b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42897 } } peers { permanent_uuid: "1ec62c6e71024161b5f1f49e6620aa3f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38761 } }
I20260929 19:03:17.908795 15674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:17.908697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34379 (local address 127.0.98.1:52583)
0929 19:03:17.908745 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:17.908773 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260929 19:03:18.333648 15285 consensus_peers.cc:487] T c97c283729124b8697ccdc0849ae101f P 1ec62c6e71024161b5f1f49e6620aa3f -> Peer a83b42620a78478f93bbddcd97803ca4 (127.0.98.3:46333): Couldn't send request to peer a83b42620a78478f93bbddcd97803ca4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c97c283729124b8697ccdc0849ae101f. This is attempt 1: this message will repeat every 5th retry.
I20260929 19:03:18.446731 15486 raft_consensus.cc:1261] T c97c283729124b8697ccdc0849ae101f P 81a3fe2385c2401cad7e2d8a240a04b0 [term 2 FOLLOWER]: Refusing update from remote peer 1ec62c6e71024161b5f1f49e6620aa3f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260929 19:03:18.447113 15673 consensus_queue.cc:1035] T c97c283729124b8697ccdc0849ae101f P 1ec62c6e71024161b5f1f49e6620aa3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "81a3fe2385c2401cad7e2d8a240a04b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260929 19:03:18.447927 15682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:18.447782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34379 (local address 127.0.98.2:36523)
0929 19:03:18.447866 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:18.447895 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76,"thread_start_us":50,"threads_started":1}
I20260929 19:03:18.448784 15686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:18.448622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34379 (local address 127.0.98.1:37107)
0929 19:03:18.448714 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:18.448740 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80,"thread_start_us":37,"threads_started":1}
I20260929 19:03:18.448868 15682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:18.448740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34379 (local address 127.0.98.2:40521)
0929 19:03:18.448771 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:18.448837 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
I20260929 19:03:18.561501 15687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:18.561311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34379 (local address 127.0.98.3:37973)
0929 19:03:18.561439 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:18.561470 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":71,"threads_started":1}
I20260929 19:03:18.775599 15688 ts_tablet_manager.cc:927] T c97c283729124b8697ccdc0849ae101f P a83b42620a78478f93bbddcd97803ca4: Initiating tablet copy from peer 1ec62c6e71024161b5f1f49e6620aa3f (127.0.98.1:38761)
I20260929 19:03:18.775866 15688 tablet_copy_client.cc:316] T c97c283729124b8697ccdc0849ae101f P a83b42620a78478f93bbddcd97803ca4: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.1:38761
Sep 29 19:03:18 dist-test-slave-lqkv krb5kdc[14101](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708596, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260929 19:03:18.783660 15690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:18.776285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43717 (local address 127.0.98.1:38761)
0929 19:03:18.776431 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:18.776435 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:18.776443 (+ 8us) server_negotiation.cc:407] Connection header received
0929 19:03:18.776465 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:18.776466 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:18.776478 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:18.776564 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:18.777123 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:18.777778 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:18.778613 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:18.778907 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:18.778987 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:18.781094 (+ 2107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:03:18.781102 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:03:18.781104 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:03:18.781116 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:03:18.782456 (+ 1340us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:18.782878 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:18.782881 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:18.782881 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:18.782951 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:18.783168 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:18.783169 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:18.783169 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:18.783301 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:03:18.783363 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:18.783510 (+ 147us) server_negotiation.cc:299] Negotiation successful
0929 19:03:18.783596 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":69,"threads_started":1}
I20260929 19:03:18.783665 15687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:18.776144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38761 (local address 127.0.98.3:43717)
0929 19:03:18.776203 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:18.776220 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:18.776289 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:18.776575 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:18.776581 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:18.776956 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:03:18.777026 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:18.777028 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:18.777866 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:18.777867 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:18.778508 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:18.778509 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:18.778983 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:18.778984 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:18.779066 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:18.779653 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:03:18.779658 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:03:18.781003 (+ 1345us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:03:18.782553 (+ 1550us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:03:18.782567 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:03:18.782569 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:03:18.782786 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:03:18.783045 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:03:18.783046 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:03:18.783046 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:03:18.783112 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:03:18.783359 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:03:18.783364 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:03:18.783407 (+ 43us) client_negotiation.cc:769] Sending connection context
0929 19:03:18.783469 (+ 62us) client_negotiation.cc:240] Negotiation successful
0929 19:03:18.783591 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260929 19:03:18.784080 15356 tablet_copy_service.cc:140] P 1ec62c6e71024161b5f1f49e6620aa3f: Received BeginTabletCopySession request for tablet c97c283729124b8697ccdc0849ae101f from peer a83b42620a78478f93bbddcd97803ca4 ({username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:43717)
I20260929 19:03:18.784154 15356 tablet_copy_service.cc:161] P 1ec62c6e71024161b5f1f49e6620aa3f: Beginning new tablet copy session on tablet c97c283729124b8697ccdc0849ae101f from peer a83b42620a78478f93bbddcd97803ca4 at {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:43717: session id = a83b42620a78478f93bbddcd97803ca4-c97c283729124b8697ccdc0849ae101f
I20260929 19:03:18.784626 15356 tablet_copy_source_session.cc:205] T c97c283729124b8697ccdc0849ae101f P 1ec62c6e71024161b5f1f49e6620aa3f: Tablet Copy: opened 0 blocks and 1 log segments
I20260929 19:03:18.785552 15688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c97c283729124b8697ccdc0849ae101f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:03:18.788079 15688 tablet_copy_client.cc:791] T c97c283729124b8697ccdc0849ae101f P a83b42620a78478f93bbddcd97803ca4: tablet copy: Starting download of 0 data blocks...
I20260929 19:03:18.788220 15688 tablet_copy_client.cc:655] T c97c283729124b8697ccdc0849ae101f P a83b42620a78478f93bbddcd97803ca4: tablet copy: Starting download of 1 WAL segments...
I20260929 19:03:18.789043 15688 tablet_copy_client.cc:531] T c97c283729124b8697ccdc0849ae101f P a83b42620a78478f93bbddcd97803ca4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260929 19:03:18.789669 15688 tablet_bootstrap.cc:492] T c97c283729124b8697ccdc0849ae101f P a83b42620a78478f93bbddcd97803ca4: Bootstrap starting.
I20260929 19:03:18.791038 15688 log.cc:824] T c97c283729124b8697ccdc0849ae101f P a83b42620a78478f93bbddcd97803ca4: Log is configured to *not* fsync() on all Append() calls
I20260929 19:03:18.793227 15688 tablet_bootstrap.cc:492] T c97c283729124b8697ccdc0849ae101f P a83b42620a78478f93bbddcd97803ca4: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 19:03:18.793455 15688 tablet_bootstrap.cc:492] T c97c283729124b8697ccdc0849ae101f P a83b42620a78478f93bbddcd97803ca4: Bootstrap complete.
I20260929 19:03:18.793588 15688 ts_tablet_manager.cc:1397] T c97c283729124b8697ccdc0849ae101f P a83b42620a78478f93bbddcd97803ca4: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20260929 19:03:18.794358 15688 raft_consensus.cc:348] T c97c283729124b8697ccdc0849ae101f P a83b42620a78478f93bbddcd97803ca4 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a83b42620a78478f93bbddcd97803ca4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46333 } } peers { permanent_uuid: "81a3fe2385c2401cad7e2d8a240a04b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42897 } } peers { permanent_uuid: "1ec62c6e71024161b5f1f49e6620aa3f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38761 } }
I20260929 19:03:18.794499 15688 raft_consensus.cc:729] T c97c283729124b8697ccdc0849ae101f P a83b42620a78478f93bbddcd97803ca4 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: a83b42620a78478f93bbddcd97803ca4, State: Initialized, Role: FOLLOWER
I20260929 19:03:18.794605 15688 consensus_queue.cc:260] T c97c283729124b8697ccdc0849ae101f P a83b42620a78478f93bbddcd97803ca4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a83b42620a78478f93bbddcd97803ca4" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46333 } } peers { permanent_uuid: "81a3fe2385c2401cad7e2d8a240a04b0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42897 } } peers { permanent_uuid: "1ec62c6e71024161b5f1f49e6620aa3f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38761 } }
I20260929 19:03:18.794824 15688 ts_tablet_manager.cc:1428] T c97c283729124b8697ccdc0849ae101f P a83b42620a78478f93bbddcd97803ca4: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260929 19:03:18.795113 15687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:18.795003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34379 (local address 127.0.98.3:39977)
0929 19:03:18.795060 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:18.795090 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260929 19:03:18.795127 15356 tablet_copy_service.cc:342] P 1ec62c6e71024161b5f1f49e6620aa3f: Request end of tablet copy session a83b42620a78478f93bbddcd97803ca4-c97c283729124b8697ccdc0849ae101f received from {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:43717
I20260929 19:03:18.795202 15356 tablet_copy_service.cc:433] P 1ec62c6e71024161b5f1f49e6620aa3f: ending tablet copy session a83b42620a78478f93bbddcd97803ca4-c97c283729124b8697ccdc0849ae101f on tablet c97c283729124b8697ccdc0849ae101f with peer a83b42620a78478f93bbddcd97803ca4
I20260929 19:03:18.795418 15687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:18.795314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34379 (local address 127.0.98.3:37611)
0929 19:03:18.795372 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:18.795397 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260929 19:03:19.169992 15694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:19.169798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34379 (local address 127.0.98.1:39801)
0929 19:03:19.169918 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:19.169948 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":69,"threads_started":1}
I20260929 19:03:19.170003 15620 raft_consensus.cc:1203] T c97c283729124b8697ccdc0849ae101f P a83b42620a78478f93bbddcd97803ca4 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 1.2->[2.3-2.3] Dedup: 2.3->[]
I20260929 19:03:19.170411 15687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:19.170321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34379 (local address 127.0.98.3:48069)
0929 19:03:19.170354 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:19.170378 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
I20260929 19:03:19.449319 15695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:19.449165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34379 (local address 127.0.98.2:35937)
0929 19:03:19.449250 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:19.449277 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74,"thread_start_us":43,"threads_started":1}
I20260929 19:03:20.170555 15696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:20.170367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34379 (local address 127.0.98.1:45537)
0929 19:03:20.170480 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:20.170511 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":56,"threads_started":1}
I20260929 19:03:20.170936 15697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:20.170774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34379 (local address 127.0.98.3:36869)
0929 19:03:20.170868 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:20.170900 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81,"thread_start_us":46,"threads_started":1}
I20260929 19:03:20.449863 15698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:20.449668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34379 (local address 127.0.98.2:53567)
0929 19:03:20.449791 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:20.449820 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":74,"threads_started":1}
I20260929 19:03:21.171201 15699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:21.170998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34379 (local address 127.0.98.1:60515)
0929 19:03:21.171119 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:21.171160 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":58,"threads_started":1}
I20260929 19:03:21.171489 15700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:21.171312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34379 (local address 127.0.98.3:33635)
0929 19:03:21.171411 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:21.171442 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85,"thread_start_us":45,"threads_started":1}
I20260929 19:03:21.450446 15701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:21.450249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34379 (local address 127.0.98.2:46219)
0929 19:03:21.450373 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:21.450408 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":57,"threads_started":1}
I20260929 19:03:22.171927 15703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:22.171735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34379 (local address 127.0.98.1:58347)
0929 19:03:22.171856 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:22.171887 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":80,"threads_started":1}
I20260929 19:03:22.172050 15704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:22.171875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34379 (local address 127.0.98.3:48279)
0929 19:03:22.171966 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:22.171991 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72,"thread_start_us":52,"threads_started":1}
I20260929 19:03:22.172240 15703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:22.172172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34379 (local address 127.0.98.1:34987)
0929 19:03:22.172203 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:22.172225 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
I20260929 19:03:22.451042 15705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:22.450877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34379 (local address 127.0.98.2:52885)
0929 19:03:22.450983 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:22.451010 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":52,"threads_started":1}
I20260929 19:03:23.172690 15706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:23.172456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34379 (local address 127.0.98.3:42951)
0929 19:03:23.172614 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:23.172646 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":105,"threads_started":1}
I20260929 19:03:23.172746 15707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:23.172566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34379 (local address 127.0.98.1:48609)
0929 19:03:23.172679 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:23.172709 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":76,"threads_started":1}
I20260929 19:03:23.451555 15708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:23.451388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34379 (local address 127.0.98.2:55993)
0929 19:03:23.451493 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:23.451524 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":55,"threads_started":1}
I20260929 19:03:24.173286 15709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:24.173094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34379 (local address 127.0.98.3:43575)
0929 19:03:24.173212 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:24.173242 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":69,"threads_started":1}
I20260929 19:03:24.173300 15710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:24.173148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34379 (local address 127.0.98.1:41883)
0929 19:03:24.173229 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:24.173263 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65,"thread_start_us":40,"threads_started":1}
I20260929 19:03:24.452167 15711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:24.451927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34379 (local address 127.0.98.2:47941)
0929 19:03:24.452101 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:24.452133 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":50,"threads_started":1}
I20260929 19:03:25.173828 15712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:25.173646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34379 (local address 127.0.98.3:34221)
0929 19:03:25.173758 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:25.173784 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":68,"threads_started":1}
I20260929 19:03:25.173844 15713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:25.173674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34379 (local address 127.0.98.1:39675)
0929 19:03:25.173781 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:25.173809 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":61,"threads_started":1}
I20260929 19:03:25.452880 15714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:25.452651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34379 (local address 127.0.98.2:39889)
0929 19:03:25.452774 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:25.452833 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":64,"threads_started":1}
I20260929 19:03:26.174376 15715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:26.174168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34379 (local address 127.0.98.3:58797)
0929 19:03:26.174286 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:26.174321 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":50,"threads_started":1}
I20260929 19:03:26.174458 15716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:26.174268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34379 (local address 127.0.98.1:41283)
0929 19:03:26.174377 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:26.174405 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":54,"threads_started":1}
I20260929 19:03:26.453370 15717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:26.453211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34379 (local address 127.0.98.2:33603)
0929 19:03:26.453310 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:26.453339 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86,"thread_start_us":51,"threads_started":1}
I20260929 19:03:27.175091 15718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:27.174870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34379 (local address 127.0.98.3:49275)
0929 19:03:27.174998 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:27.175030 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":87,"threads_started":1}
I20260929 19:03:27.175099 15719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:27.174916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34379 (local address 127.0.98.1:56175)
0929 19:03:27.175014 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:27.175045 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84,"thread_start_us":59,"threads_started":1}
I20260929 19:03:27.453924 15720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:27.453755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34379 (local address 127.0.98.2:60985)
0929 19:03:27.453861 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:27.453893 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":67,"threads_started":1}
I20260929 19:03:28.175778 15722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:28.175585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34379 (local address 127.0.98.1:50717)
0929 19:03:28.175710 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:28.175735 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":81,"threads_started":1}
I20260929 19:03:28.175810 15723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:28.175639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34379 (local address 127.0.98.3:42407)
0929 19:03:28.175748 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:28.175773 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":62,"threads_started":1}
I20260929 19:03:28.454535 15724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:28.454373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34379 (local address 127.0.98.2:40557)
0929 19:03:28.454475 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:28.454503 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":57,"threads_started":1}
I20260929 19:03:29.176425 15725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:29.176239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34379 (local address 127.0.98.1:55347)
0929 19:03:29.176355 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:29.176382 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":72,"threads_started":1}
I20260929 19:03:29.176509 15726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:29.176312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34379 (local address 127.0.98.3:34363)
0929 19:03:29.176423 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:29.176450 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":64,"threads_started":1}
I20260929 19:03:29.455116 15727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:29.454964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34379 (local address 127.0.98.2:57297)
0929 19:03:29.455058 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:29.455086 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82,"thread_start_us":53,"threads_started":1}
I20260929 19:03:30.177153 15728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:30.176966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34379 (local address 127.0.98.1:44691)
0929 19:03:30.177084 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:30.177111 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":72,"threads_started":1}
I20260929 19:03:30.177227 15729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:30.177022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34379 (local address 127.0.98.3:46921)
0929 19:03:30.177137 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:30.177169 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":72,"threads_started":1}
I20260929 19:03:30.455652 15730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:30.455481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34379 (local address 127.0.98.2:49409)
0929 19:03:30.455581 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:30.455607 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":52,"threads_started":1}
I20260929 19:03:31.177706 15731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:31.177490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34379 (local address 127.0.98.1:60381)
0929 19:03:31.177608 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:31.177660 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":66,"threads_started":1}
I20260929 19:03:31.177774 15732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:31.177535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34379 (local address 127.0.98.3:59829)
0929 19:03:31.177677 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:31.177714 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":104,"threads_started":1}
I20260929 19:03:31.456430 15733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:31.456239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34379 (local address 127.0.98.2:42723)
0929 19:03:31.456357 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:31.456400 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34379: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":52,"threads_started":1}
I20260929 19:03:31.576680 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUbnLHE/build/release/bin/kudu with pid 15239
I20260929 19:03:31.581382 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUbnLHE/build/release/bin/kudu with pid 15401
I20260929 19:03:31.585207 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUbnLHE/build/release/bin/kudu with pid 15535
2026-09-29T19:03:31Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (17947 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 19:03:31 dist-test-slave-lqkv krb5kdc[15736](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 19:03:31 dist-test-slave-lqkv krb5kdc[15736](info): set up 2 sockets
Sep 29 19:03:31 dist-test-slave-lqkv krb5kdc[15736](info): commencing operation
krb5kdc: starting...
W20260929 19:03:32.548738 392 mini_kdc.cc:121] Time spent starting KDC: real 0.944s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/krb5kdc/test-user.keytab.
Sep 29 19:03:32 dist-test-slave-lqkv krb5kdc[15736](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708612, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T19:03:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T19:03:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 19:03:32.653314 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUbnLHE/build/release/bin/kudu
/tmp/dist-test-taskUbnLHE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40647
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40647
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:03:32.730450 15749 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:03:32.730629 15749 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:03:32.730650 15749 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:03:32.731720 15749 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260929 19:03:32.731791 15749 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260929 19:03:32.731803 15749 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260929 19:03:32.731906 15749 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 19:03:32.731979 15749 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:03:32.732023 15749 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:03:32.732036 15749 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 19:03:32.732082 15749 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 19:03:32.733708 15749 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40647
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40647
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15749
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 18:09:13 UTC on ccycloud.quasar-hpkalf.root.comops.site
build id 4680913
I20260929 19:03:32.733953 15749 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:03:32.734166 15749 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:03:32.738420 15757 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:03:32.738423 15754 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:03:32.738456 15755 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:03:32.738469 15749 server_base.cc:1034] running on GCE node
I20260929 19:03:32.738796 15749 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:03:32.739066 15749 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:03:32.740223 15749 hybrid_clock.cc:648] HybridClock initialized: now 1790708612740214 us; error 36 us; skew 500 ppm
Sep 29 19:03:32 dist-test-slave-lqkv krb5kdc[15736](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708612, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:03:32.742590 15749 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 19:03:32.742930 15749 webserver.cc:460] Webserver started at http://127.0.98.62:34769/ using document root <none> and password file <none>
I20260929 19:03:32.743095 15749 fs_manager.cc:359] Metadata directory not provided
I20260929 19:03:32.743132 15749 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:03:32.743211 15749 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:03:32.744117 15749 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/minicluster-data/master-0/data/instance:
uuid: "37a91a7695be4407b44d568eae618e7b"
format_stamp: "Formatted at 2026-09-29 19:03:32 on dist-test-slave-lqkv"
server_key: "65e3240cbb4140095e76fd16e533939a"
server_key_iv: "6e817c256fb43c0a2df9f04fa59f5a39"
server_key_version: "encryptionkey@0"
I20260929 19:03:32.744397 15749 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/minicluster-data/master-0/wal/instance:
uuid: "37a91a7695be4407b44d568eae618e7b"
format_stamp: "Formatted at 2026-09-29 19:03:32 on dist-test-slave-lqkv"
server_key: "65e3240cbb4140095e76fd16e533939a"
server_key_iv: "6e817c256fb43c0a2df9f04fa59f5a39"
server_key_version: "encryptionkey@0"
I20260929 19:03:32.745544 15749 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 19:03:32.746134 15764 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:32.746299 15749 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 19:03:32.746362 15749 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/minicluster-data/master-0/wal
uuid: "37a91a7695be4407b44d568eae618e7b"
format_stamp: "Formatted at 2026-09-29 19:03:32 on dist-test-slave-lqkv"
server_key: "65e3240cbb4140095e76fd16e533939a"
server_key_iv: "6e817c256fb43c0a2df9f04fa59f5a39"
server_key_version: "encryptionkey@0"
I20260929 19:03:32.746403 15749 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:03:32.760737 15749 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:03:32.762817 15749 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:03:32.762940 15749 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:03:32.766449 15749 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40647
I20260929 19:03:32.766510 15816 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40647 every 8 connection(s)
I20260929 19:03:32.766847 15749 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/minicluster-data/master-0/data/info.pb
I20260929 19:03:32.766997 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUbnLHE/build/release/bin/kudu as pid 15749
I20260929 19:03:32.767123 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/minicluster-data/master-0/wal/instance
I20260929 19:03:32.767326 392 external_mini_cluster.cc:1428] Setting key 4fc90e26916b6a23745cd73ccf19b9b0
I20260929 19:03:32.767786 15817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:03:32.770190 15817 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37a91a7695be4407b44d568eae618e7b: Bootstrap starting.
I20260929 19:03:32.770744 15817 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 37a91a7695be4407b44d568eae618e7b: Neither blocks nor log segments found. Creating new log.
I20260929 19:03:32.771090 15817 log.cc:824] T 00000000000000000000000000000000 P 37a91a7695be4407b44d568eae618e7b: Log is configured to *not* fsync() on all Append() calls
I20260929 19:03:32.771776 15817 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37a91a7695be4407b44d568eae618e7b: No bootstrap required, opened a new log
Sep 29 19:03:32 dist-test-slave-lqkv krb5kdc[15736](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708612, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 19:03:32.773895 15817 raft_consensus.cc:348] T 00000000000000000000000000000000 P 37a91a7695be4407b44d568eae618e7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37a91a7695be4407b44d568eae618e7b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40647 } }
I20260929 19:03:32.774065 15817 raft_consensus.cc:374] T 00000000000000000000000000000000 P 37a91a7695be4407b44d568eae618e7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:03:32.774092 15817 raft_consensus.cc:729] T 00000000000000000000000000000000 P 37a91a7695be4407b44d568eae618e7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37a91a7695be4407b44d568eae618e7b, State: Initialized, Role: FOLLOWER
I20260929 19:03:32.774243 15817 consensus_queue.cc:260] T 00000000000000000000000000000000 P 37a91a7695be4407b44d568eae618e7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37a91a7695be4407b44d568eae618e7b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40647 } }
I20260929 19:03:32.774353 15817 raft_consensus.cc:388] T 00000000000000000000000000000000 P 37a91a7695be4407b44d568eae618e7b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 19:03:32.774396 15817 raft_consensus.cc:482] T 00000000000000000000000000000000 P 37a91a7695be4407b44d568eae618e7b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 19:03:32.774456 15817 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 37a91a7695be4407b44d568eae618e7b [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:03:32.775024 15817 raft_consensus.cc:504] T 00000000000000000000000000000000 P 37a91a7695be4407b44d568eae618e7b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37a91a7695be4407b44d568eae618e7b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40647 } }
I20260929 19:03:32.775149 15817 leader_election.cc:302] T 00000000000000000000000000000000 P 37a91a7695be4407b44d568eae618e7b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 37a91a7695be4407b44d568eae618e7b; no voters:
I20260929 19:03:32.775314 15817 leader_election.cc:288] T 00000000000000000000000000000000 P 37a91a7695be4407b44d568eae618e7b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 19:03:32.775345 15822 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 37a91a7695be4407b44d568eae618e7b [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:03:32.775478 15822 raft_consensus.cc:686] T 00000000000000000000000000000000 P 37a91a7695be4407b44d568eae618e7b [term 1 LEADER]: Becoming Leader. State: Replica: 37a91a7695be4407b44d568eae618e7b, State: Running, Role: LEADER
I20260929 19:03:32.775586 15817 sys_catalog.cc:564] T 00000000000000000000000000000000 P 37a91a7695be4407b44d568eae618e7b [sys.catalog]: configured and running, proceeding with master startup.
I20260929 19:03:32.775609 15822 consensus_queue.cc:237] T 00000000000000000000000000000000 P 37a91a7695be4407b44d568eae618e7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37a91a7695be4407b44d568eae618e7b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40647 } }
I20260929 19:03:32.775947 15822 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37a91a7695be4407b44d568eae618e7b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 37a91a7695be4407b44d568eae618e7b. Latest consensus state: current_term: 1 leader_uuid: "37a91a7695be4407b44d568eae618e7b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37a91a7695be4407b44d568eae618e7b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40647 } } }
I20260929 19:03:32.775965 15824 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37a91a7695be4407b44d568eae618e7b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "37a91a7695be4407b44d568eae618e7b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37a91a7695be4407b44d568eae618e7b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40647 } } }
I20260929 19:03:32.776132 15824 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37a91a7695be4407b44d568eae618e7b [sys.catalog]: This master's current role is: LEADER
I20260929 19:03:32.776120 15822 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37a91a7695be4407b44d568eae618e7b [sys.catalog]: This master's current role is: LEADER
I20260929 19:03:32.776674 15831 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 19:03:32.777264 15831 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 19:03:32.777665 15820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:32.767814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38628 (local address 127.0.98.62:40647)
0929 19:03:32.768014 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:32.768022 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:32.768050 (+ 28us) server_negotiation.cc:407] Connection header received
0929 19:03:32.768410 (+ 360us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:32.768415 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:32.768454 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:32.768587 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:32.769230 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:32.770087 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:32.770916 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:32.771463 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:32.771581 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:32.774005 (+ 2424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:03:32.774008 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:03:32.774008 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:03:32.774017 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:03:32.775969 (+ 1952us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:32.776383 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:32.776385 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:32.776385 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:32.776474 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:32.776680 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:32.776681 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:32.776681 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:32.777088 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:03:32.777271 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:32.777338 (+ 67us) server_negotiation.cc:299] Negotiation successful
0929 19:03:32.777480 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":91,"threads_started":1}
I20260929 19:03:32.778834 15831 catalog_manager.cc:1347] Generated new cluster ID: 487198941692428cb04133850e10e3ad
I20260929 19:03:32.778913 15831 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 19:03:32.794673 15831 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 19:03:32.795174 15831 catalog_manager.cc:1504] Loading token signing keys...
I20260929 19:03:32.802670 15831 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 37a91a7695be4407b44d568eae618e7b: Generated new TSK 0
I20260929 19:03:32.802829 15831 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260929 19:03:32.833420 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUbnLHE/build/release/bin/kudu
/tmp/dist-test-taskUbnLHE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40647
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:03:32.906486 15843 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:03:32.906672 15843 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:03:32.906708 15843 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:03:32.907778 15843 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:03:32.907812 15843 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:03:32.907836 15843 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 19:03:32.909106 15843 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40647
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15843
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 18:09:13 UTC on ccycloud.quasar-hpkalf.root.comops.site
build id 4680913
I20260929 19:03:32.909281 15843 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:03:32.909430 15843 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:03:32.913790 15851 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:03:32.913767 15849 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:03:32.913816 15848 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:03:32.914021 15843 server_base.cc:1034] running on GCE node
I20260929 19:03:32.914207 15843 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:03:32.914371 15843 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:03:32.915520 15843 hybrid_clock.cc:648] HybridClock initialized: now 1790708612915504 us; error 28 us; skew 500 ppm
Sep 29 19:03:32 dist-test-slave-lqkv krb5kdc[15736](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708612, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:03:32.917855 15843 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 19:03:32.918241 15843 webserver.cc:460] Webserver started at http://127.0.98.1:41849/ using document root <none> and password file <none>
I20260929 19:03:32.918443 15843 fs_manager.cc:359] Metadata directory not provided
I20260929 19:03:32.918507 15843 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:03:32.918644 15843 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:03:32.919591 15843 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/minicluster-data/ts-0/data/instance:
uuid: "45494feaa6df4f32807cb11e2b10a536"
format_stamp: "Formatted at 2026-09-29 19:03:32 on dist-test-slave-lqkv"
server_key: "a7f246d56390ed72651d42d8a5f3c919"
server_key_iv: "0d2f63a0d7078e0d0ae5e7d52d9e9635"
server_key_version: "encryptionkey@0"
I20260929 19:03:32.919924 15843 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/minicluster-data/ts-0/wal/instance:
uuid: "45494feaa6df4f32807cb11e2b10a536"
format_stamp: "Formatted at 2026-09-29 19:03:32 on dist-test-slave-lqkv"
server_key: "a7f246d56390ed72651d42d8a5f3c919"
server_key_iv: "0d2f63a0d7078e0d0ae5e7d52d9e9635"
server_key_version: "encryptionkey@0"
I20260929 19:03:32.921182 15843 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 19:03:32.921770 15858 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:32.921945 15843 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 19:03:32.922034 15843 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/minicluster-data/ts-0/wal
uuid: "45494feaa6df4f32807cb11e2b10a536"
format_stamp: "Formatted at 2026-09-29 19:03:32 on dist-test-slave-lqkv"
server_key: "a7f246d56390ed72651d42d8a5f3c919"
server_key_iv: "0d2f63a0d7078e0d0ae5e7d52d9e9635"
server_key_version: "encryptionkey@0"
I20260929 19:03:32.922096 15843 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:03:32.931245 15843 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:03:32.932776 15843 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:03:32.932967 15843 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:03:32.933173 15843 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:03:32.933435 15843 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:03:32.933485 15843 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:32.933516 15843 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:03:32.933553 15843 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:32.938763 15843 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42445
I20260929 19:03:32.938793 15971 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42445 every 8 connection(s)
I20260929 19:03:32.939103 15843 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/minicluster-data/ts-0/data/info.pb
Sep 29 19:03:32 dist-test-slave-lqkv krb5kdc[15736](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708612, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 19:03:32.946439 15820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:32.939740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35503 (local address 127.0.98.62:40647)
0929 19:03:32.939800 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:32.939803 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:32.939982 (+ 179us) server_negotiation.cc:407] Connection header received
0929 19:03:32.940370 (+ 388us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:32.940371 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:32.940380 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:32.940428 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:32.941271 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:32.941714 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:32.942532 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:32.942703 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:32.942745 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:32.944385 (+ 1640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:03:32.944386 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:03:32.944386 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:03:32.944391 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:03:32.945481 (+ 1090us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:32.945795 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:32.945796 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:32.945796 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:32.945839 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:32.946006 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:32.946006 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:32.946006 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:32.946090 (+ 84us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:03:32.946133 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:32.946319 (+ 186us) server_negotiation.cc:299] Negotiation successful
0929 19:03:32.946383 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260929 19:03:32.946563 15974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:32.939719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40647 (local address 127.0.98.1:35503)
0929 19:03:32.939874 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:32.939905 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:32.940291 (+ 386us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:32.940473 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:32.940476 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:32.941050 (+ 574us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:03:32.941205 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:32.941209 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:32.941782 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:32.941783 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:32.942468 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:32.942470 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:32.942764 (+ 294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:32.942765 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:32.942851 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:32.943201 (+ 350us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:03:32.943207 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:03:32.944307 (+ 1100us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:03:32.945543 (+ 1236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:03:32.945548 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:03:32.945549 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:03:32.945738 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:03:32.945885 (+ 147us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:03:32.945885 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:03:32.945885 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:03:32.945960 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:03:32.946141 (+ 181us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:03:32.946143 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:03:32.946229 (+ 86us) client_negotiation.cc:769] Sending connection context
0929 19:03:32.946295 (+ 66us) client_negotiation.cc:240] Negotiation successful
0929 19:03:32.946394 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":84,"threads_started":1}
I20260929 19:03:32.947026 15972 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40647
I20260929 19:03:32.947096 15972 heartbeater.cc:464] Registering TS with master...
I20260929 19:03:32.947259 15972 heartbeater.cc:511] Master 127.0.98.62:40647 requested a full tablet report, sending...
I20260929 19:03:32.947301 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUbnLHE/build/release/bin/kudu as pid 15843
I20260929 19:03:32.947412 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/minicluster-data/ts-0/wal/instance
I20260929 19:03:32.947639 392 external_mini_cluster.cc:1428] Setting key 8dd86cff49bac7584f3768f28fd9e333
I20260929 19:03:32.947795 15781 ts_manager.cc:194] Registered new tserver with Master: 45494feaa6df4f32807cb11e2b10a536 (127.0.98.1:42445)
I20260929 19:03:32.948774 15781 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:35503
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260929 19:03:32.969146 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUbnLHE/build/release/bin/kudu
/tmp/dist-test-taskUbnLHE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40647
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:03:33.043491 15977 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:03:33.043710 15977 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:03:33.043751 15977 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:03:33.044797 15977 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:03:33.044884 15977 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:03:33.044929 15977 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 19:03:33.046267 15977 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40647
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15977
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 18:09:13 UTC on ccycloud.quasar-hpkalf.root.comops.site
build id 4680913
I20260929 19:03:33.046527 15977 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:03:33.046737 15977 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:03:33.051535 15985 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:03:33.051575 15983 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:03:33.051548 15982 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:03:33.051828 15977 server_base.cc:1034] running on GCE node
I20260929 19:03:33.052026 15977 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:03:33.052255 15977 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:03:33.053418 15977 hybrid_clock.cc:648] HybridClock initialized: now 1790708613053406 us; error 27 us; skew 500 ppm
Sep 29 19:03:33 dist-test-slave-lqkv krb5kdc[15736](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708613, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:03:33.055745 15977 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 19:03:33.056180 15977 webserver.cc:460] Webserver started at http://127.0.98.2:37303/ using document root <none> and password file <none>
I20260929 19:03:33.056378 15977 fs_manager.cc:359] Metadata directory not provided
I20260929 19:03:33.056420 15977 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:03:33.056572 15977 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:03:33.057552 15977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/minicluster-data/ts-1/data/instance:
uuid: "c8150c87f2684d4c937a086a3ef65e2a"
format_stamp: "Formatted at 2026-09-29 19:03:33 on dist-test-slave-lqkv"
server_key: "e7b0ad82b16a7d630e34d8eebaf9958f"
server_key_iv: "076a095a7e7f74a0625173fc5676be78"
server_key_version: "encryptionkey@0"
I20260929 19:03:33.057890 15977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/minicluster-data/ts-1/wal/instance:
uuid: "c8150c87f2684d4c937a086a3ef65e2a"
format_stamp: "Formatted at 2026-09-29 19:03:33 on dist-test-slave-lqkv"
server_key: "e7b0ad82b16a7d630e34d8eebaf9958f"
server_key_iv: "076a095a7e7f74a0625173fc5676be78"
server_key_version: "encryptionkey@0"
I20260929 19:03:33.059083 15977 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260929 19:03:33.059769 15992 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:33.059932 15977 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 19:03:33.060012 15977 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/minicluster-data/ts-1/wal
uuid: "c8150c87f2684d4c937a086a3ef65e2a"
format_stamp: "Formatted at 2026-09-29 19:03:33 on dist-test-slave-lqkv"
server_key: "e7b0ad82b16a7d630e34d8eebaf9958f"
server_key_iv: "076a095a7e7f74a0625173fc5676be78"
server_key_version: "encryptionkey@0"
I20260929 19:03:33.060062 15977 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:03:33.075490 15977 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:03:33.077258 15977 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:03:33.077353 15977 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:03:33.077489 15977 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:03:33.077742 15977 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:03:33.077771 15977 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:33.077790 15977 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:03:33.077826 15977 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:33.082959 15977 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35361
I20260929 19:03:33.083024 16105 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35361 every 8 connection(s)
I20260929 19:03:33.083282 15977 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/minicluster-data/ts-1/data/info.pb
Sep 29 19:03:33 dist-test-slave-lqkv krb5kdc[15736](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708613, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 19:03:33.091301 15820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:33.083882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47065 (local address 127.0.98.62:40647)
0929 19:03:33.083929 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:33.083931 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:33.084113 (+ 182us) server_negotiation.cc:407] Connection header received
0929 19:03:33.084562 (+ 449us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:33.084563 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:33.084586 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:33.084630 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:33.085467 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:33.085928 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:33.086858 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:33.087052 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:33.087130 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:33.089068 (+ 1938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:03:33.089069 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:03:33.089069 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:03:33.089074 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:03:33.090235 (+ 1161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:33.090574 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:33.090575 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:33.090575 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:33.090622 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:33.090829 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:33.090830 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:33.090830 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:33.090937 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:03:33.090982 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:33.091189 (+ 207us) server_negotiation.cc:299] Negotiation successful
0929 19:03:33.091245 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260929 19:03:33.091444 16108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:33.083858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40647 (local address 127.0.98.2:47065)
0929 19:03:33.084006 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:33.084042 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:33.084457 (+ 415us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:33.084686 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:33.084689 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:33.085209 (+ 520us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:03:33.085365 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:33.085368 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:33.086019 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:33.086020 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:33.086724 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:33.086726 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:33.087139 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:33.087140 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:33.087263 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:33.087707 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:03:33.087715 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:03:33.088994 (+ 1279us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:03:33.090302 (+ 1308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:03:33.090308 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:03:33.090309 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:03:33.090510 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:03:33.090679 (+ 169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:03:33.090680 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:03:33.090680 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:03:33.090767 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:03:33.090993 (+ 226us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:03:33.090997 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:03:33.091086 (+ 89us) client_negotiation.cc:769] Sending connection context
0929 19:03:33.091161 (+ 75us) client_negotiation.cc:240] Negotiation successful
0929 19:03:33.091258 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":71,"threads_started":1}
I20260929 19:03:33.091905 16106 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40647
I20260929 19:03:33.091974 16106 heartbeater.cc:464] Registering TS with master...
I20260929 19:03:33.092159 16106 heartbeater.cc:511] Master 127.0.98.62:40647 requested a full tablet report, sending...
I20260929 19:03:33.092518 15781 ts_manager.cc:194] Registered new tserver with Master: c8150c87f2684d4c937a086a3ef65e2a (127.0.98.2:35361)
I20260929 19:03:33.092926 15781 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:47065
I20260929 19:03:33.093060 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUbnLHE/build/release/bin/kudu as pid 15977
I20260929 19:03:33.093171 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/minicluster-data/ts-1/wal/instance
I20260929 19:03:33.093381 392 external_mini_cluster.cc:1428] Setting key cd9a87a89b405749241ef2c490d3bfa5
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260929 19:03:33.115371 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUbnLHE/build/release/bin/kudu
/tmp/dist-test-taskUbnLHE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40647
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:03:33.188418 16111 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:03:33.188655 16111 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:03:33.188692 16111 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:03:33.189821 16111 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:03:33.189886 16111 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:03:33.189941 16111 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 19:03:33.191294 16111 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40647
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16111
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 18:09:13 UTC on ccycloud.quasar-hpkalf.root.comops.site
build id 4680913
I20260929 19:03:33.191524 16111 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:03:33.191722 16111 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:03:33.196022 16117 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:03:33.196153 16111 server_base.cc:1034] running on GCE node
W20260929 19:03:33.196024 16119 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:03:33.196079 16116 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:03:33.196476 16111 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:03:33.196765 16111 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:03:33.197933 16111 hybrid_clock.cc:648] HybridClock initialized: now 1790708613197927 us; error 30 us; skew 500 ppm
Sep 29 19:03:33 dist-test-slave-lqkv krb5kdc[15736](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708613, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:03:33.200152 16111 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 19:03:33.200546 16111 webserver.cc:460] Webserver started at http://127.0.98.3:43749/ using document root <none> and password file <none>
I20260929 19:03:33.200764 16111 fs_manager.cc:359] Metadata directory not provided
I20260929 19:03:33.200877 16111 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:03:33.201026 16111 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:03:33.201927 16111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/minicluster-data/ts-2/data/instance:
uuid: "43c7b519591f4119a105016fef7c4656"
format_stamp: "Formatted at 2026-09-29 19:03:33 on dist-test-slave-lqkv"
server_key: "9dbbaaf51b9418e023cc82e84ca5e550"
server_key_iv: "c339eb06ae9f36bf9a77ec5eb2974423"
server_key_version: "encryptionkey@0"
I20260929 19:03:33.202253 16111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/minicluster-data/ts-2/wal/instance:
uuid: "43c7b519591f4119a105016fef7c4656"
format_stamp: "Formatted at 2026-09-29 19:03:33 on dist-test-slave-lqkv"
server_key: "9dbbaaf51b9418e023cc82e84ca5e550"
server_key_iv: "c339eb06ae9f36bf9a77ec5eb2974423"
server_key_version: "encryptionkey@0"
I20260929 19:03:33.203457 16111 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260929 19:03:33.204123 16126 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:33.204276 16111 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 19:03:33.204370 16111 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/minicluster-data/ts-2/wal
uuid: "43c7b519591f4119a105016fef7c4656"
format_stamp: "Formatted at 2026-09-29 19:03:33 on dist-test-slave-lqkv"
server_key: "9dbbaaf51b9418e023cc82e84ca5e550"
server_key_iv: "c339eb06ae9f36bf9a77ec5eb2974423"
server_key_version: "encryptionkey@0"
I20260929 19:03:33.204447 16111 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:03:33.228642 16111 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:03:33.230494 16111 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:03:33.230655 16111 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:03:33.230895 16111 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:03:33.231210 16111 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:03:33.231261 16111 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:33.231297 16111 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:03:33.231319 16111 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:33.236778 16111 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44775
I20260929 19:03:33.236886 16239 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44775 every 8 connection(s)
I20260929 19:03:33.237176 16111 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/minicluster-data/ts-2/data/info.pb
I20260929 19:03:33.239260 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUbnLHE/build/release/bin/kudu as pid 16111
I20260929 19:03:33.239375 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/minicluster-data/ts-2/wal/instance
I20260929 19:03:33.239540 392 external_mini_cluster.cc:1428] Setting key b79180df31be32ca09e6a8c2668fcf7a
Sep 29 19:03:33 dist-test-slave-lqkv krb5kdc[15736](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708613, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 19:03:33.245934 15820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:33.238039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52231 (local address 127.0.98.62:40647)
0929 19:03:33.238130 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:33.238132 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:33.238274 (+ 142us) server_negotiation.cc:407] Connection header received
0929 19:03:33.238786 (+ 512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:33.238787 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:33.238799 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:33.238844 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:33.239865 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:33.240443 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:33.241510 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:33.241696 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:33.241760 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:33.243572 (+ 1812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:03:33.243574 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:03:33.243575 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:03:33.243581 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:03:33.244671 (+ 1090us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:33.245082 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:33.245083 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:33.245083 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:33.245138 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:33.245361 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:33.245362 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:33.245362 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:33.245465 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:03:33.245531 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:33.245782 (+ 251us) server_negotiation.cc:299] Negotiation successful
0929 19:03:33.245852 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 19:03:33.246119 16242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:33.237933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40647 (local address 127.0.98.3:52231)
0929 19:03:33.238140 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:33.238190 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:33.238681 (+ 491us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:33.238908 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:33.238914 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:33.239517 (+ 603us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:03:33.239776 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:33.239784 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:33.240540 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:33.240541 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:33.241415 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:33.241417 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:33.241777 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:33.241778 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:33.241892 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:33.242271 (+ 379us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:03:33.242278 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:03:33.243485 (+ 1207us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:03:33.244762 (+ 1277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:03:33.244768 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:03:33.244770 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:03:33.244999 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:03:33.245201 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:03:33.245202 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:03:33.245202 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:03:33.245299 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:03:33.245531 (+ 232us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:03:33.245537 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:03:33.245657 (+ 120us) client_negotiation.cc:769] Sending connection context
0929 19:03:33.245748 (+ 91us) client_negotiation.cc:240] Negotiation successful
0929 19:03:33.245897 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":91,"threads_started":1}
I20260929 19:03:33.246515 16240 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40647
I20260929 19:03:33.246596 16240 heartbeater.cc:464] Registering TS with master...
I20260929 19:03:33.246718 16240 heartbeater.cc:511] Master 127.0.98.62:40647 requested a full tablet report, sending...
I20260929 19:03:33.247071 15781 ts_manager.cc:194] Registered new tserver with Master: 43c7b519591f4119a105016fef7c4656 (127.0.98.3:44775)
I20260929 19:03:33.247426 15781 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:52231
I20260929 19:03:33.251310 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 19:03:33.258024 15820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:33.252505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38630 (local address 127.0.98.62:40647)
0929 19:03:33.252560 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:33.252562 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:33.252667 (+ 105us) server_negotiation.cc:407] Connection header received
0929 19:03:33.252733 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:33.252734 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:33.252743 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:33.252876 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:33.253374 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:33.253777 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:33.254530 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:33.254712 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:33.254778 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:33.255807 (+ 1029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:03:33.255808 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:03:33.255809 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:03:33.255815 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:03:33.256909 (+ 1094us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:33.257251 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:33.257252 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:33.257253 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:33.257318 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:33.257510 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:33.257511 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:33.257511 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:33.257644 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:03:33.257742 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:33.257858 (+ 116us) server_negotiation.cc:299] Negotiation successful
0929 19:03:33.257929 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260929 19:03:33.259505 15781 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38630:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 19:03:33.261098 15781 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 19:03:33.266976 16251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:33.263070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42445 (local address 127.0.0.1:36282)
0929 19:03:33.263396 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:33.263406 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:33.263458 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:33.264011 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:33.264012 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:33.264014 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:03:33.264062 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:33.264063 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:33.264917 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:33.264918 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:33.265576 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:33.265577 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:33.266644 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:33.266645 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:33.266728 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:33.266740 (+ 12us) client_negotiation.cc:769] Sending connection context
0929 19:03:33.266827 (+ 87us) client_negotiation.cc:240] Negotiation successful
0929 19:03:33.266899 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"mutex_wait_us":3,"thread_start_us":58,"threads_started":1}
I20260929 19:03:33.267005 16252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:33.263418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36282 (local address 127.0.98.1:42445)
0929 19:03:33.263737 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:33.263741 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:33.263751 (+ 10us) server_negotiation.cc:407] Connection header received
0929 19:03:33.263791 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:33.263793 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:33.263807 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:33.263998 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:03:33.264142 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:33.264772 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:33.265661 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:33.266535 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:33.266607 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:33.266675 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:33.266874 (+ 199us) server_negotiation.cc:299] Negotiation successful
0929 19:03:33.266927 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":90,"threads_started":1}
I20260929 19:03:33.267041 16250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:33.263133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57356 (local address 127.0.98.3:44775)
0929 19:03:33.263307 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:33.263313 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:33.263328 (+ 15us) server_negotiation.cc:407] Connection header received
0929 19:03:33.263419 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:33.263422 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:33.263438 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:33.263555 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:03:33.263792 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:33.264564 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:33.265944 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:33.266549 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:33.266604 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:33.266675 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:33.266924 (+ 249us) server_negotiation.cc:299] Negotiation successful
0929 19:03:33.266964 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":88,"threads_started":1}
I20260929 19:03:33.267227 16248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:33.263070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44775 (local address 127.0.0.1:57356)
0929 19:03:33.263227 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:33.263261 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:33.263319 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:33.263656 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:33.263659 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:33.263662 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:03:33.263720 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:33.263721 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:33.264660 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:33.264662 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:33.265868 (+ 1206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:33.265870 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:33.266710 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:33.266711 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:33.266789 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:33.266793 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 19:03:33.267105 (+ 312us) client_negotiation.cc:240] Negotiation successful
0929 19:03:33.267157 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":63,"threads_started":1}
I20260929 19:03:33.268070 16249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:33.263259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35361 (local address 127.0.0.1:43658)
0929 19:03:33.263449 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:33.263459 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:33.263497 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:33.264895 (+ 1398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:33.264895 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:33.264898 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:03:33.264936 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:33.264937 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:33.266090 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:33.266091 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:33.266841 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:33.266843 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:33.267814 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:33.267815 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:33.267884 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:33.267888 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 19:03:33.267950 (+ 62us) client_negotiation.cc:240] Negotiation successful
0929 19:03:33.267998 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"mutex_wait_us":2,"thread_start_us":79,"threads_started":1}
I20260929 19:03:33.268215 16253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:33.263697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43658 (local address 127.0.98.2:35361)
0929 19:03:33.264729 (+ 1032us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:33.264733 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:33.264741 (+ 8us) server_negotiation.cc:407] Connection header received
0929 19:03:33.264775 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:33.264777 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:33.264790 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:33.264998 (+ 208us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:03:33.265061 (+ 63us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:33.266016 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:33.266951 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:33.267722 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:33.267794 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:33.267846 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:33.267991 (+ 145us) server_negotiation.cc:299] Negotiation successful
0929 19:03:33.268033 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1001,"thread_start_us":118,"threads_started":1}
I20260929 19:03:33.268115 16174 tablet_service.cc:1467] Processing CreateTablet for tablet c7782e56046a47539036f874ace86f91 (DEFAULT_TABLE table=test-table [id=749c6d60b5ee421f867df493103e1a5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:03:33.268138 15906 tablet_service.cc:1467] Processing CreateTablet for tablet c7782e56046a47539036f874ace86f91 (DEFAULT_TABLE table=test-table [id=749c6d60b5ee421f867df493103e1a5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:03:33.268476 16174 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7782e56046a47539036f874ace86f91. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:03:33.268477 15906 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7782e56046a47539036f874ace86f91. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:03:33.268720 16040 tablet_service.cc:1467] Processing CreateTablet for tablet c7782e56046a47539036f874ace86f91 (DEFAULT_TABLE table=test-table [id=749c6d60b5ee421f867df493103e1a5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:03:33.269021 16040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7782e56046a47539036f874ace86f91. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:03:33.271100 16254 tablet_bootstrap.cc:492] T c7782e56046a47539036f874ace86f91 P 43c7b519591f4119a105016fef7c4656: Bootstrap starting.
I20260929 19:03:33.271662 16254 tablet_bootstrap.cc:654] T c7782e56046a47539036f874ace86f91 P 43c7b519591f4119a105016fef7c4656: Neither blocks nor log segments found. Creating new log.
I20260929 19:03:33.271871 16255 tablet_bootstrap.cc:492] T c7782e56046a47539036f874ace86f91 P 45494feaa6df4f32807cb11e2b10a536: Bootstrap starting.
I20260929 19:03:33.272253 16254 log.cc:824] T c7782e56046a47539036f874ace86f91 P 43c7b519591f4119a105016fef7c4656: Log is configured to *not* fsync() on all Append() calls
I20260929 19:03:33.272264 16256 tablet_bootstrap.cc:492] T c7782e56046a47539036f874ace86f91 P c8150c87f2684d4c937a086a3ef65e2a: Bootstrap starting.
I20260929 19:03:33.272563 16255 tablet_bootstrap.cc:654] T c7782e56046a47539036f874ace86f91 P 45494feaa6df4f32807cb11e2b10a536: Neither blocks nor log segments found. Creating new log.
I20260929 19:03:33.272912 16256 tablet_bootstrap.cc:654] T c7782e56046a47539036f874ace86f91 P c8150c87f2684d4c937a086a3ef65e2a: Neither blocks nor log segments found. Creating new log.
I20260929 19:03:33.273172 16254 tablet_bootstrap.cc:492] T c7782e56046a47539036f874ace86f91 P 43c7b519591f4119a105016fef7c4656: No bootstrap required, opened a new log
I20260929 19:03:33.273176 16255 log.cc:824] T c7782e56046a47539036f874ace86f91 P 45494feaa6df4f32807cb11e2b10a536: Log is configured to *not* fsync() on all Append() calls
I20260929 19:03:33.273245 16254 ts_tablet_manager.cc:1397] T c7782e56046a47539036f874ace86f91 P 43c7b519591f4119a105016fef7c4656: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 19:03:33.273448 16256 log.cc:824] T c7782e56046a47539036f874ace86f91 P c8150c87f2684d4c937a086a3ef65e2a: Log is configured to *not* fsync() on all Append() calls
I20260929 19:03:33.273811 16255 tablet_bootstrap.cc:492] T c7782e56046a47539036f874ace86f91 P 45494feaa6df4f32807cb11e2b10a536: No bootstrap required, opened a new log
I20260929 19:03:33.273900 16255 ts_tablet_manager.cc:1397] T c7782e56046a47539036f874ace86f91 P 45494feaa6df4f32807cb11e2b10a536: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 19:03:33.274174 16256 tablet_bootstrap.cc:492] T c7782e56046a47539036f874ace86f91 P c8150c87f2684d4c937a086a3ef65e2a: No bootstrap required, opened a new log
I20260929 19:03:33.274246 16256 ts_tablet_manager.cc:1397] T c7782e56046a47539036f874ace86f91 P c8150c87f2684d4c937a086a3ef65e2a: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260929 19:03:33.274895 16254 raft_consensus.cc:348] T c7782e56046a47539036f874ace86f91 P 43c7b519591f4119a105016fef7c4656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43c7b519591f4119a105016fef7c4656" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44775 } } peers { permanent_uuid: "c8150c87f2684d4c937a086a3ef65e2a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35361 } } peers { permanent_uuid: "45494feaa6df4f32807cb11e2b10a536" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42445 } }
I20260929 19:03:33.275151 16254 raft_consensus.cc:374] T c7782e56046a47539036f874ace86f91 P 43c7b519591f4119a105016fef7c4656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:03:33.275206 16254 raft_consensus.cc:729] T c7782e56046a47539036f874ace86f91 P 43c7b519591f4119a105016fef7c4656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43c7b519591f4119a105016fef7c4656, State: Initialized, Role: FOLLOWER
I20260929 19:03:33.275338 16254 consensus_queue.cc:260] T c7782e56046a47539036f874ace86f91 P 43c7b519591f4119a105016fef7c4656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43c7b519591f4119a105016fef7c4656" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44775 } } peers { permanent_uuid: "c8150c87f2684d4c937a086a3ef65e2a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35361 } } peers { permanent_uuid: "45494feaa6df4f32807cb11e2b10a536" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42445 } }
I20260929 19:03:33.275592 16254 ts_tablet_manager.cc:1428] T c7782e56046a47539036f874ace86f91 P 43c7b519591f4119a105016fef7c4656: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 19:03:33.275728 16240 heartbeater.cc:503] Master 127.0.98.62:40647 was elected leader, sending a full tablet report...
I20260929 19:03:33.276113 16256 raft_consensus.cc:348] T c7782e56046a47539036f874ace86f91 P c8150c87f2684d4c937a086a3ef65e2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43c7b519591f4119a105016fef7c4656" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44775 } } peers { permanent_uuid: "c8150c87f2684d4c937a086a3ef65e2a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35361 } } peers { permanent_uuid: "45494feaa6df4f32807cb11e2b10a536" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42445 } }
I20260929 19:03:33.276113 16255 raft_consensus.cc:348] T c7782e56046a47539036f874ace86f91 P 45494feaa6df4f32807cb11e2b10a536 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43c7b519591f4119a105016fef7c4656" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44775 } } peers { permanent_uuid: "c8150c87f2684d4c937a086a3ef65e2a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35361 } } peers { permanent_uuid: "45494feaa6df4f32807cb11e2b10a536" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42445 } }
I20260929 19:03:33.276268 16255 raft_consensus.cc:374] T c7782e56046a47539036f874ace86f91 P 45494feaa6df4f32807cb11e2b10a536 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:03:33.276268 16256 raft_consensus.cc:374] T c7782e56046a47539036f874ace86f91 P c8150c87f2684d4c937a086a3ef65e2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:03:33.276307 16256 raft_consensus.cc:729] T c7782e56046a47539036f874ace86f91 P c8150c87f2684d4c937a086a3ef65e2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8150c87f2684d4c937a086a3ef65e2a, State: Initialized, Role: FOLLOWER
I20260929 19:03:33.276307 16255 raft_consensus.cc:729] T c7782e56046a47539036f874ace86f91 P 45494feaa6df4f32807cb11e2b10a536 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45494feaa6df4f32807cb11e2b10a536, State: Initialized, Role: FOLLOWER
I20260929 19:03:33.276402 16255 consensus_queue.cc:260] T c7782e56046a47539036f874ace86f91 P 45494feaa6df4f32807cb11e2b10a536 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43c7b519591f4119a105016fef7c4656" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44775 } } peers { permanent_uuid: "c8150c87f2684d4c937a086a3ef65e2a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35361 } } peers { permanent_uuid: "45494feaa6df4f32807cb11e2b10a536" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42445 } }
I20260929 19:03:33.276448 16256 consensus_queue.cc:260] T c7782e56046a47539036f874ace86f91 P c8150c87f2684d4c937a086a3ef65e2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43c7b519591f4119a105016fef7c4656" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44775 } } peers { permanent_uuid: "c8150c87f2684d4c937a086a3ef65e2a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35361 } } peers { permanent_uuid: "45494feaa6df4f32807cb11e2b10a536" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42445 } }
I20260929 19:03:33.276652 16255 ts_tablet_manager.cc:1428] T c7782e56046a47539036f874ace86f91 P 45494feaa6df4f32807cb11e2b10a536: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260929 19:03:33.276674 16256 ts_tablet_manager.cc:1428] T c7782e56046a47539036f874ace86f91 P c8150c87f2684d4c937a086a3ef65e2a: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260929 19:03:33.276754 16106 heartbeater.cc:503] Master 127.0.98.62:40647 was elected leader, sending a full tablet report...
I20260929 19:03:33.276754 15972 heartbeater.cc:503] Master 127.0.98.62:40647 was elected leader, sending a full tablet report...
W20260929 19:03:33.333920 16107 tablet.cc:2369] T c7782e56046a47539036f874ace86f91 P c8150c87f2684d4c937a086a3ef65e2a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 19:03:33.439757 15973 tablet.cc:2369] T c7782e56046a47539036f874ace86f91 P 45494feaa6df4f32807cb11e2b10a536: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 19:03:33.487739 16241 tablet.cc:2369] T c7782e56046a47539036f874ace86f91 P 43c7b519591f4119a105016fef7c4656: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 19:03:33.496500 16262 raft_consensus.cc:482] T c7782e56046a47539036f874ace86f91 P c8150c87f2684d4c937a086a3ef65e2a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 19:03:33.496634 16262 raft_consensus.cc:504] T c7782e56046a47539036f874ace86f91 P c8150c87f2684d4c937a086a3ef65e2a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43c7b519591f4119a105016fef7c4656" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44775 } } peers { permanent_uuid: "c8150c87f2684d4c937a086a3ef65e2a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35361 } } peers { permanent_uuid: "45494feaa6df4f32807cb11e2b10a536" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42445 } }
I20260929 19:03:33.497071 16262 leader_election.cc:288] T c7782e56046a47539036f874ace86f91 P c8150c87f2684d4c937a086a3ef65e2a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 43c7b519591f4119a105016fef7c4656 (127.0.98.3:44775), 45494feaa6df4f32807cb11e2b10a536 (127.0.98.1:42445)
I20260929 19:03:33.500012 16252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:33.497532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34797 (local address 127.0.98.1:42445)
0929 19:03:33.497588 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:33.497590 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:33.497597 (+ 7us) server_negotiation.cc:407] Connection header received
0929 19:03:33.497629 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:33.497630 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:33.497640 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:33.497706 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:03:33.497881 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:33.498345 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:33.499303 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:33.499650 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:33.499751 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:33.499762 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:33.499910 (+ 148us) server_negotiation.cc:299] Negotiation successful
0929 19:03:33.499947 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260929 19:03:33.500026 16263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:33.497283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42445 (local address 127.0.98.2:34797)
0929 19:03:33.497484 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:33.497494 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:33.497538 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:33.497710 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:33.497711 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:33.497712 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:03:33.497798 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:33.497800 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:33.498452 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:33.498453 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:33.499211 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:33.499213 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:33.499753 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:33.499754 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:33.499824 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:33.499828 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 19:03:33.499903 (+ 75us) client_negotiation.cc:240] Negotiation successful
0929 19:03:33.499942 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":127,"threads_started":1}
I20260929 19:03:33.500470 15926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7782e56046a47539036f874ace86f91" candidate_uuid: "c8150c87f2684d4c937a086a3ef65e2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45494feaa6df4f32807cb11e2b10a536" is_pre_election: true
I20260929 19:03:33.500630 15926 raft_consensus.cc:2454] T c7782e56046a47539036f874ace86f91 P 45494feaa6df4f32807cb11e2b10a536 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8150c87f2684d4c937a086a3ef65e2a in term 0.
I20260929 19:03:33.500901 15994 leader_election.cc:302] T c7782e56046a47539036f874ace86f91 P c8150c87f2684d4c937a086a3ef65e2a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45494feaa6df4f32807cb11e2b10a536, c8150c87f2684d4c937a086a3ef65e2a; no voters:
I20260929 19:03:33.501026 16262 raft_consensus.cc:2781] T c7782e56046a47539036f874ace86f91 P c8150c87f2684d4c937a086a3ef65e2a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 19:03:33.501067 16262 raft_consensus.cc:482] T c7782e56046a47539036f874ace86f91 P c8150c87f2684d4c937a086a3ef65e2a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 19:03:33.501093 16262 raft_consensus.cc:3037] T c7782e56046a47539036f874ace86f91 P c8150c87f2684d4c937a086a3ef65e2a [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:03:33.501224 16108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:33.497271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44775 (local address 127.0.98.2:34017)
0929 19:03:33.497327 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:33.497339 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:33.497381 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:33.497684 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:33.497686 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:33.497688 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:03:33.497763 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:33.497766 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:33.498775 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:33.498776 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:33.500071 (+ 1295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:33.500074 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:33.500975 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:33.500976 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:33.501053 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:33.501056 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 19:03:33.501115 (+ 59us) client_negotiation.cc:240] Negotiation successful
0929 19:03:33.501149 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260929 19:03:33.501266 16250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:33.497321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34017 (local address 127.0.98.3:44775)
0929 19:03:33.497497 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:33.497500 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:33.497511 (+ 11us) server_negotiation.cc:407] Connection header received
0929 19:03:33.497543 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:33.497544 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:33.497557 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:33.497628 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:03:33.497838 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:33.498673 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:33.500175 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:33.500908 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:33.500976 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:33.500986 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:33.501128 (+ 142us) server_negotiation.cc:299] Negotiation successful
0929 19:03:33.501173 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146}
I20260929 19:03:33.501956 16262 raft_consensus.cc:504] T c7782e56046a47539036f874ace86f91 P c8150c87f2684d4c937a086a3ef65e2a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43c7b519591f4119a105016fef7c4656" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44775 } } peers { permanent_uuid: "c8150c87f2684d4c937a086a3ef65e2a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35361 } } peers { permanent_uuid: "45494feaa6df4f32807cb11e2b10a536" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42445 } }
I20260929 19:03:33.502244 15926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7782e56046a47539036f874ace86f91" candidate_uuid: "c8150c87f2684d4c937a086a3ef65e2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45494feaa6df4f32807cb11e2b10a536"
I20260929 19:03:33.502328 15926 raft_consensus.cc:3037] T c7782e56046a47539036f874ace86f91 P 45494feaa6df4f32807cb11e2b10a536 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:03:33.502434 16193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7782e56046a47539036f874ace86f91" candidate_uuid: "c8150c87f2684d4c937a086a3ef65e2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43c7b519591f4119a105016fef7c4656"
I20260929 19:03:33.502442 16194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7782e56046a47539036f874ace86f91" candidate_uuid: "c8150c87f2684d4c937a086a3ef65e2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43c7b519591f4119a105016fef7c4656" is_pre_election: true
I20260929 19:03:33.502545 16193 raft_consensus.cc:3037] T c7782e56046a47539036f874ace86f91 P 43c7b519591f4119a105016fef7c4656 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:03:33.502185 16262 leader_election.cc:288] T c7782e56046a47539036f874ace86f91 P c8150c87f2684d4c937a086a3ef65e2a [CANDIDATE]: Term 1 election: Requested vote from peers 43c7b519591f4119a105016fef7c4656 (127.0.98.3:44775), 45494feaa6df4f32807cb11e2b10a536 (127.0.98.1:42445)
I20260929 19:03:33.503221 15926 raft_consensus.cc:2454] T c7782e56046a47539036f874ace86f91 P 45494feaa6df4f32807cb11e2b10a536 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8150c87f2684d4c937a086a3ef65e2a in term 1.
I20260929 19:03:33.503351 16193 raft_consensus.cc:2454] T c7782e56046a47539036f874ace86f91 P 43c7b519591f4119a105016fef7c4656 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8150c87f2684d4c937a086a3ef65e2a in term 1.
I20260929 19:03:33.503366 15994 leader_election.cc:302] T c7782e56046a47539036f874ace86f91 P c8150c87f2684d4c937a086a3ef65e2a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45494feaa6df4f32807cb11e2b10a536, c8150c87f2684d4c937a086a3ef65e2a; no voters:
I20260929 19:03:33.503530 16262 raft_consensus.cc:2781] T c7782e56046a47539036f874ace86f91 P c8150c87f2684d4c937a086a3ef65e2a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:03:33.503693 16262 raft_consensus.cc:686] T c7782e56046a47539036f874ace86f91 P c8150c87f2684d4c937a086a3ef65e2a [term 1 LEADER]: Becoming Leader. State: Replica: c8150c87f2684d4c937a086a3ef65e2a, State: Running, Role: LEADER
I20260929 19:03:33.503826 16262 consensus_queue.cc:237] T c7782e56046a47539036f874ace86f91 P c8150c87f2684d4c937a086a3ef65e2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43c7b519591f4119a105016fef7c4656" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44775 } } peers { permanent_uuid: "c8150c87f2684d4c937a086a3ef65e2a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35361 } } peers { permanent_uuid: "45494feaa6df4f32807cb11e2b10a536" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42445 } }
I20260929 19:03:33.504779 15781 catalog_manager.cc:5495] T c7782e56046a47539036f874ace86f91 P c8150c87f2684d4c937a086a3ef65e2a reported cstate change: term changed from 0 to 1, leader changed from <none> to c8150c87f2684d4c937a086a3ef65e2a (127.0.98.2). New cstate: current_term: 1 leader_uuid: "c8150c87f2684d4c937a086a3ef65e2a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43c7b519591f4119a105016fef7c4656" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8150c87f2684d4c937a086a3ef65e2a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "45494feaa6df4f32807cb11e2b10a536" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42445 } health_report { overall_health: UNKNOWN } } }
I20260929 19:03:33.550673 16253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:33.548202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43666 (local address 127.0.98.2:35361)
0929 19:03:33.548250 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:33.548252 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:33.548261 (+ 9us) server_negotiation.cc:407] Connection header received
0929 19:03:33.548317 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:33.548317 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:33.548336 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:33.548402 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 19:03:33.548533 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:33.549206 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:33.550036 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:33.550221 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:33.550285 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:33.550393 (+ 108us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 19:03:33.550490 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 19:03:33.550546 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:33.550579 (+ 33us) server_negotiation.cc:299] Negotiation successful
0929 19:03:33.550597 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260929 19:03:33.552537 16193 raft_consensus.cc:1261] T c7782e56046a47539036f874ace86f91 P 43c7b519591f4119a105016fef7c4656 [term 1 FOLLOWER]: Refusing update from remote peer c8150c87f2684d4c937a086a3ef65e2a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 19:03:33.552583 15926 raft_consensus.cc:1261] T c7782e56046a47539036f874ace86f91 P 45494feaa6df4f32807cb11e2b10a536 [term 1 FOLLOWER]: Refusing update from remote peer c8150c87f2684d4c937a086a3ef65e2a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 19:03:33.552866 16262 consensus_queue.cc:1035] T c7782e56046a47539036f874ace86f91 P c8150c87f2684d4c937a086a3ef65e2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "43c7b519591f4119a105016fef7c4656" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 19:03:33.553041 16264 consensus_queue.cc:1035] T c7782e56046a47539036f874ace86f91 P c8150c87f2684d4c937a086a3ef65e2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "45494feaa6df4f32807cb11e2b10a536" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 19:03:33.555346 16267 mvcc.cc:205] Tried to move back new op lower bound from 7334742481108545536 to 7334742480912277504. Current Snapshot: MvccSnapshot[applied={T|T < 7334742481108545536}]
I20260929 19:03:33.556219 16268 mvcc.cc:205] Tried to move back new op lower bound from 7334742481108545536 to 7334742480912277504. Current Snapshot: MvccSnapshot[applied={T|T < 7334742481108545536 or (T in {7334742481108545536})}]
I20260929 19:03:33.557520 15780 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38630:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 19:03:33.557610 15780 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38630:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 19:03:33.558341 15780 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 37a91a7695be4407b44d568eae618e7b: Sending DeleteTablet for 3 replicas of tablet c7782e56046a47539036f874ace86f91
I20260929 19:03:33.558619 16174 tablet_service.cc:1514] Processing DeleteTablet for tablet c7782e56046a47539036f874ace86f91 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:03:33 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57356
I20260929 19:03:33.558645 15906 tablet_service.cc:1514] Processing DeleteTablet for tablet c7782e56046a47539036f874ace86f91 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:03:33 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36282
I20260929 19:03:33.558672 16040 tablet_service.cc:1514] Processing DeleteTablet for tablet c7782e56046a47539036f874ace86f91 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:03:33 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43658
I20260929 19:03:33.558879 16278 tablet_replica.cc:330] T c7782e56046a47539036f874ace86f91 P c8150c87f2684d4c937a086a3ef65e2a: stopping tablet replica
I20260929 19:03:33.558946 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUbnLHE/build/release/bin/kudu with pid 15843
I20260929 19:03:33.559028 16278 raft_consensus.cc:2229] T c7782e56046a47539036f874ace86f91 P c8150c87f2684d4c937a086a3ef65e2a [term 1 LEADER]: Raft consensus shutting down.
I20260929 19:03:33.559113 16278 raft_consensus.cc:2258] T c7782e56046a47539036f874ace86f91 P c8150c87f2684d4c937a086a3ef65e2a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 19:03:33.560063 16278 ts_tablet_manager.cc:1905] T c7782e56046a47539036f874ace86f91 P c8150c87f2684d4c937a086a3ef65e2a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 19:03:33.561900 16278 ts_tablet_manager.cc:1918] T c7782e56046a47539036f874ace86f91 P c8150c87f2684d4c937a086a3ef65e2a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 19:03:33.562034 16278 log.cc:1196] T c7782e56046a47539036f874ace86f91 P c8150c87f2684d4c937a086a3ef65e2a: Deleting WAL directory at /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/minicluster-data/ts-1/wal/wals/c7782e56046a47539036f874ace86f91
I20260929 19:03:33.562345 16278 ts_tablet_manager.cc:1939] T c7782e56046a47539036f874ace86f91 P c8150c87f2684d4c937a086a3ef65e2a: Deleting consensus metadata
I20260929 19:03:33.562783 15767 catalog_manager.cc:4841] TS c8150c87f2684d4c937a086a3ef65e2a (127.0.98.2:35361): tablet c7782e56046a47539036f874ace86f91 (table test-table [id=749c6d60b5ee421f867df493103e1a5b]) successfully deleted
W20260929 19:03:33.564221 15766 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:42445 (error 108)
I20260929 19:03:33.564458 16277 tablet_replica.cc:330] T c7782e56046a47539036f874ace86f91 P 43c7b519591f4119a105016fef7c4656: stopping tablet replica
I20260929 19:03:33.564571 16277 raft_consensus.cc:2229] T c7782e56046a47539036f874ace86f91 P 43c7b519591f4119a105016fef7c4656 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 19:03:33.564713 16249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:33.564579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42445 (local address 127.0.0.1:36296)
0929 19:03:33.564624 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:33.564665 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42445: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260929 19:03:33.564756 16277 raft_consensus.cc:2258] T c7782e56046a47539036f874ace86f91 P 43c7b519591f4119a105016fef7c4656 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260929 19:03:33.564958 15766 catalog_manager.cc:4584] TS 45494feaa6df4f32807cb11e2b10a536 (127.0.98.1:42445): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c7782e56046a47539036f874ace86f91: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42445: connect: Connection refused (error 111)
I20260929 19:03:33.564996 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUbnLHE/build/release/bin/kudu with pid 15977
I20260929 19:03:33.566660 16277 ts_tablet_manager.cc:1905] T c7782e56046a47539036f874ace86f91 P 43c7b519591f4119a105016fef7c4656: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 19:03:33.568617 16277 ts_tablet_manager.cc:1918] T c7782e56046a47539036f874ace86f91 P 43c7b519591f4119a105016fef7c4656: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 19:03:33.568717 16277 log.cc:1196] T c7782e56046a47539036f874ace86f91 P 43c7b519591f4119a105016fef7c4656: Deleting WAL directory at /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/minicluster-data/ts-2/wal/wals/c7782e56046a47539036f874ace86f91
I20260929 19:03:33.569341 16277 ts_tablet_manager.cc:1939] T c7782e56046a47539036f874ace86f91 P 43c7b519591f4119a105016fef7c4656: Deleting consensus metadata
I20260929 19:03:33.569774 15767 catalog_manager.cc:4841] TS 43c7b519591f4119a105016fef7c4656 (127.0.98.3:44775): tablet c7782e56046a47539036f874ace86f91 (table test-table [id=749c6d60b5ee421f867df493103e1a5b]) successfully deleted
I20260929 19:03:33.571355 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUbnLHE/build/release/bin/kudu with pid 16111
I20260929 19:03:33.575927 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUbnLHE/build/release/bin/kudu with pid 15749
I20260929 19:03:53.581064 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUbnLHE/build/release/bin/kudu
/tmp/dist-test-taskUbnLHE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40647
--webserver_interface=127.0.98.62
--webserver_port=34769
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40647
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:03:53.656128 16281 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:03:53.656297 16281 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:03:53.656318 16281 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:03:53.657451 16281 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260929 19:03:53.657490 16281 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260929 19:03:53.657502 16281 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260929 19:03:53.657514 16281 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 19:03:53.657527 16281 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:03:53.657580 16281 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:03:53.657604 16281 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 19:03:53.657631 16281 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 19:03:53.659018 16281 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40647
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40647
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=34769
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16281
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 18:09:13 UTC on ccycloud.quasar-hpkalf.root.comops.site
build id 4680913
I20260929 19:03:53.659255 16281 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:03:53.659474 16281 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:03:53.669346 16288 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:03:53.669384 16290 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:03:53.669353 16287 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:03:53.669603 16281 server_base.cc:1034] running on GCE node
I20260929 19:03:53.669839 16281 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:03:53.670147 16281 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:03:53.671283 16281 hybrid_clock.cc:648] HybridClock initialized: now 1790708633671251 us; error 42 us; skew 500 ppm
Sep 29 19:03:53 dist-test-slave-lqkv krb5kdc[15736](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708633, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:03:53.677919 16281 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 19:03:53.678347 16281 webserver.cc:460] Webserver started at http://127.0.98.62:34769/ using document root <none> and password file <none>
I20260929 19:03:53.678542 16281 fs_manager.cc:359] Metadata directory not provided
I20260929 19:03:53.678589 16281 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:03:53.679823 16281 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 19:03:53.680397 16297 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:53.680577 16281 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:53.680661 16281 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/minicluster-data/master-0/wal
uuid: "37a91a7695be4407b44d568eae618e7b"
format_stamp: "Formatted at 2026-09-29 19:03:32 on dist-test-slave-lqkv"
server_key: "65e3240cbb4140095e76fd16e533939a"
server_key_iv: "6e817c256fb43c0a2df9f04fa59f5a39"
server_key_version: "encryptionkey@0"
I20260929 19:03:53.681046 16281 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:03:53.689796 16281 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:03:53.691344 16281 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:03:53.691440 16281 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:03:53.695124 16281 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40647
I20260929 19:03:53.695152 16349 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40647 every 8 connection(s)
I20260929 19:03:53.695533 16281 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/minicluster-data/master-0/data/info.pb
I20260929 19:03:53.695776 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUbnLHE/build/release/bin/kudu as pid 16281
I20260929 19:03:53.696115 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUbnLHE/build/release/bin/kudu
/tmp/dist-test-taskUbnLHE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:42445
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:40647
--webserver_port=41849
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 19:03:53.697561 16350 sys_catalog.cc:263] Verifying existing consensus state
I20260929 19:03:53.698182 16350 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37a91a7695be4407b44d568eae618e7b: Bootstrap starting.
I20260929 19:03:53.706331 16350 log.cc:824] T 00000000000000000000000000000000 P 37a91a7695be4407b44d568eae618e7b: Log is configured to *not* fsync() on all Append() calls
I20260929 19:03:53.708004 16350 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37a91a7695be4407b44d568eae618e7b: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 19:03:53.708295 16350 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37a91a7695be4407b44d568eae618e7b: Bootstrap complete.
I20260929 19:03:53.710472 16350 raft_consensus.cc:348] T 00000000000000000000000000000000 P 37a91a7695be4407b44d568eae618e7b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37a91a7695be4407b44d568eae618e7b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40647 } }
I20260929 19:03:53.710690 16350 raft_consensus.cc:729] T 00000000000000000000000000000000 P 37a91a7695be4407b44d568eae618e7b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37a91a7695be4407b44d568eae618e7b, State: Initialized, Role: FOLLOWER
I20260929 19:03:53.710843 16350 consensus_queue.cc:260] T 00000000000000000000000000000000 P 37a91a7695be4407b44d568eae618e7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37a91a7695be4407b44d568eae618e7b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40647 } }
I20260929 19:03:53.710897 16350 raft_consensus.cc:388] T 00000000000000000000000000000000 P 37a91a7695be4407b44d568eae618e7b [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 19:03:53.710954 16350 raft_consensus.cc:482] T 00000000000000000000000000000000 P 37a91a7695be4407b44d568eae618e7b [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 19:03:53.710999 16350 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 37a91a7695be4407b44d568eae618e7b [term 1 FOLLOWER]: Advancing to term 2
I20260929 19:03:53.711931 16350 raft_consensus.cc:504] T 00000000000000000000000000000000 P 37a91a7695be4407b44d568eae618e7b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37a91a7695be4407b44d568eae618e7b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40647 } }
I20260929 19:03:53.712009 16350 leader_election.cc:302] T 00000000000000000000000000000000 P 37a91a7695be4407b44d568eae618e7b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 37a91a7695be4407b44d568eae618e7b; no voters:
I20260929 19:03:53.712220 16350 leader_election.cc:288] T 00000000000000000000000000000000 P 37a91a7695be4407b44d568eae618e7b [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 19:03:53.712251 16354 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 37a91a7695be4407b44d568eae618e7b [term 2 FOLLOWER]: Leader election won for term 2
I20260929 19:03:53.712363 16354 raft_consensus.cc:686] T 00000000000000000000000000000000 P 37a91a7695be4407b44d568eae618e7b [term 2 LEADER]: Becoming Leader. State: Replica: 37a91a7695be4407b44d568eae618e7b, State: Running, Role: LEADER
I20260929 19:03:53.712471 16350 sys_catalog.cc:564] T 00000000000000000000000000000000 P 37a91a7695be4407b44d568eae618e7b [sys.catalog]: configured and running, proceeding with master startup.
I20260929 19:03:53.712507 16354 consensus_queue.cc:237] T 00000000000000000000000000000000 P 37a91a7695be4407b44d568eae618e7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37a91a7695be4407b44d568eae618e7b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40647 } }
I20260929 19:03:53.712914 16355 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37a91a7695be4407b44d568eae618e7b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "37a91a7695be4407b44d568eae618e7b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37a91a7695be4407b44d568eae618e7b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40647 } } }
I20260929 19:03:53.713017 16355 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37a91a7695be4407b44d568eae618e7b [sys.catalog]: This master's current role is: LEADER
I20260929 19:03:53.713207 16356 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37a91a7695be4407b44d568eae618e7b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 37a91a7695be4407b44d568eae618e7b. Latest consensus state: current_term: 2 leader_uuid: "37a91a7695be4407b44d568eae618e7b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37a91a7695be4407b44d568eae618e7b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40647 } } }
I20260929 19:03:53.713299 16356 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37a91a7695be4407b44d568eae618e7b [sys.catalog]: This master's current role is: LEADER
I20260929 19:03:53.713433 16361 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 19:03:53.714180 16361 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 19:03:53.714272 16361 catalog_manager.cc:1259] Loaded cluster ID: 487198941692428cb04133850e10e3ad
I20260929 19:03:53.714344 16361 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 19:03:53.715219 16361 catalog_manager.cc:1504] Loading token signing keys...
I20260929 19:03:53.715406 16361 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 37a91a7695be4407b44d568eae618e7b: Loaded TSK: 0
I20260929 19:03:53.728905 16361 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 37a91a7695be4407b44d568eae618e7b: Generated new TSK 1
I20260929 19:03:53.729080 16361 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 19:03:53.782430 16352 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:03:53.782634 16352 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:03:53.782672 16352 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:03:53.783838 16352 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:03:53.783897 16352 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:03:53.783938 16352 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 19:03:53.785499 16352 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:42445
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=41849
--tserver_master_addrs=127.0.98.62:40647
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16352
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 18:09:13 UTC on ccycloud.quasar-hpkalf.root.comops.site
build id 4680913
I20260929 19:03:53.785735 16352 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:03:53.785941 16352 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:03:53.791546 16377 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:03:53.791546 16378 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:03:53.791667 16380 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:03:53.791936 16352 server_base.cc:1034] running on GCE node
I20260929 19:03:53.792164 16352 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:03:53.792404 16352 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:03:53.793561 16352 hybrid_clock.cc:648] HybridClock initialized: now 1790708633793547 us; error 33 us; skew 500 ppm
Sep 29 19:03:53 dist-test-slave-lqkv krb5kdc[15736](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708633, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:03:53.795998 16352 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 19:03:53.796434 16352 webserver.cc:460] Webserver started at http://127.0.98.1:41849/ using document root <none> and password file <none>
I20260929 19:03:53.796643 16352 fs_manager.cc:359] Metadata directory not provided
I20260929 19:03:53.796730 16352 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:03:53.798020 16352 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260929 19:03:53.798663 16387 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:53.798832 16352 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 19:03:53.798928 16352 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/minicluster-data/ts-0/wal
uuid: "45494feaa6df4f32807cb11e2b10a536"
format_stamp: "Formatted at 2026-09-29 19:03:32 on dist-test-slave-lqkv"
server_key: "a7f246d56390ed72651d42d8a5f3c919"
server_key_iv: "0d2f63a0d7078e0d0ae5e7d52d9e9635"
server_key_version: "encryptionkey@0"
I20260929 19:03:53.799218 16352 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:03:53.807289 16352 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:03:53.809232 16352 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:03:53.809361 16352 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:03:53.809585 16352 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:03:53.810042 16394 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260929 19:03:53.811345 16352 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260929 19:03:53.811393 16352 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.001s user 0.000s sys 0.000s
I20260929 19:03:53.811424 16352 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260929 19:03:53.812029 16352 ts_tablet_manager.cc:610] Registered 1 tablets
I20260929 19:03:53.812073 16352 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260929 19:03:53.812134 16394 tablet_bootstrap.cc:492] T c7782e56046a47539036f874ace86f91 P 45494feaa6df4f32807cb11e2b10a536: Bootstrap starting.
I20260929 19:03:53.819451 16352 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42445
I20260929 19:03:53.819515 16501 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42445 every 8 connection(s)
I20260929 19:03:53.819970 16352 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/minicluster-data/ts-0/data/info.pb
I20260929 19:03:53.820999 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUbnLHE/build/release/bin/kudu as pid 16352
I20260929 19:03:53.821362 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUbnLHE/build/release/bin/kudu
/tmp/dist-test-taskUbnLHE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:35361
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:40647
--webserver_port=37303
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 19:03:53.823603 16394 log.cc:824] T c7782e56046a47539036f874ace86f91 P 45494feaa6df4f32807cb11e2b10a536: Log is configured to *not* fsync() on all Append() calls
I20260929 19:03:53.826956 16394 tablet_bootstrap.cc:492] T c7782e56046a47539036f874ace86f91 P 45494feaa6df4f32807cb11e2b10a536: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 19:03:53.827334 16394 tablet_bootstrap.cc:492] T c7782e56046a47539036f874ace86f91 P 45494feaa6df4f32807cb11e2b10a536: Bootstrap complete.
I20260929 19:03:53.828888 16394 ts_tablet_manager.cc:1397] T c7782e56046a47539036f874ace86f91 P 45494feaa6df4f32807cb11e2b10a536: Time spent bootstrapping tablet: real 0.017s user 0.011s sys 0.004s
Sep 29 19:03:53 dist-test-slave-lqkv krb5kdc[15736](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708633, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 19:03:53.830158 16394 raft_consensus.cc:348] T c7782e56046a47539036f874ace86f91 P 45494feaa6df4f32807cb11e2b10a536 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43c7b519591f4119a105016fef7c4656" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44775 } } peers { permanent_uuid: "c8150c87f2684d4c937a086a3ef65e2a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35361 } } peers { permanent_uuid: "45494feaa6df4f32807cb11e2b10a536" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42445 } }
I20260929 19:03:53.830780 16394 raft_consensus.cc:729] T c7782e56046a47539036f874ace86f91 P 45494feaa6df4f32807cb11e2b10a536 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45494feaa6df4f32807cb11e2b10a536, State: Initialized, Role: FOLLOWER
I20260929 19:03:53.830956 16394 consensus_queue.cc:260] T c7782e56046a47539036f874ace86f91 P 45494feaa6df4f32807cb11e2b10a536 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43c7b519591f4119a105016fef7c4656" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44775 } } peers { permanent_uuid: "c8150c87f2684d4c937a086a3ef65e2a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35361 } } peers { permanent_uuid: "45494feaa6df4f32807cb11e2b10a536" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42445 } }
I20260929 19:03:53.831557 16394 ts_tablet_manager.cc:1428] T c7782e56046a47539036f874ace86f91 P 45494feaa6df4f32807cb11e2b10a536: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260929 19:03:53.834301 16504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:53.820601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40647 (local address 127.0.98.1:41505)
0929 19:03:53.821174 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:53.821217 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:53.821767 (+ 550us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:53.822755 (+ 988us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:53.822759 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:53.823124 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:03:53.823357 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:53.823363 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:53.824711 (+ 1348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:53.824712 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:53.826081 (+ 1369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:53.826084 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:53.827063 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:53.827065 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:53.827236 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:53.827832 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:03:53.827843 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:03:53.829957 (+ 2114us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:03:53.832407 (+ 2450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:03:53.832414 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:03:53.832415 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:03:53.832709 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:03:53.833073 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:03:53.833074 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:03:53.833075 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:03:53.833203 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:03:53.833677 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:03:53.833683 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:03:53.833824 (+ 141us) client_negotiation.cc:769] Sending connection context
0929 19:03:53.833932 (+ 108us) client_negotiation.cc:240] Negotiation successful
0929 19:03:53.834109 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":118,"threads_started":1}
I20260929 19:03:53.834344 16505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:53.821002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41505 (local address 127.0.98.62:40647)
0929 19:03:53.821963 (+ 961us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:53.821972 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:53.822016 (+ 44us) server_negotiation.cc:407] Connection header received
0929 19:03:53.822137 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:53.822143 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:53.822194 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:53.822318 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:53.823498 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:53.824566 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:53.826198 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:53.826911 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:53.827522 (+ 611us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:53.830098 (+ 2576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:03:53.830102 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:03:53.830103 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:03:53.830114 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:03:53.832309 (+ 2195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:53.832857 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:53.832860 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:53.832861 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:53.832978 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:53.833293 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:53.833294 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:53.833295 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:53.833586 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:03:53.833677 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:53.834030 (+ 353us) server_negotiation.cc:299] Negotiation successful
0929 19:03:53.834151 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":916,"thread_start_us":71,"threads_started":1}
I20260929 19:03:53.835157 16502 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40647
I20260929 19:03:53.835239 16502 heartbeater.cc:464] Registering TS with master...
I20260929 19:03:53.835398 16502 heartbeater.cc:511] Master 127.0.98.62:40647 requested a full tablet report, sending...
I20260929 19:03:53.836091 16314 ts_manager.cc:194] Registered new tserver with Master: 45494feaa6df4f32807cb11e2b10a536 (127.0.98.1:42445)
I20260929 19:03:53.837196 16314 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:41505
I20260929 19:03:53.838126 16502 heartbeater.cc:503] Master 127.0.98.62:40647 was elected leader, sending a full tablet report...
I20260929 19:03:53.850023 16510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:53.836839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37420 (local address 127.0.98.1:42445)
0929 19:03:53.846327 (+ 9488us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:53.846331 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:53.846340 (+ 9us) server_negotiation.cc:407] Connection header received
0929 19:03:53.846370 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:53.846372 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:53.846386 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:53.846463 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:03:53.846636 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:53.847640 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:53.849039 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:53.849673 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:53.849748 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:53.849796 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:53.849927 (+ 131us) server_negotiation.cc:299] Negotiation successful
0929 19:03:53.849965 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9459,"thread_start_us":63,"threads_started":1}
I20260929 19:03:53.850031 16509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:53.836628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42445 (local address 127.0.0.1:37420)
0929 19:03:53.844927 (+ 8299us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:53.844970 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:53.845020 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:53.846500 (+ 1480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:53.846502 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:53.846505 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:03:53.846556 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:53.846557 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:53.847723 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:53.847724 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:53.848954 (+ 1230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:53.848957 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:53.849755 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:53.849755 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:53.849830 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:53.849841 (+ 11us) client_negotiation.cc:769] Sending connection context
0929 19:03:53.849904 (+ 63us) client_negotiation.cc:240] Negotiation successful
0929 19:03:53.849947 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8277,"thread_start_us":68,"threads_started":1}
I20260929 19:03:53.850589 16418 tablet_service.cc:1514] Processing DeleteTablet for tablet c7782e56046a47539036f874ace86f91 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:03:33 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37420
I20260929 19:03:53.851050 16511 tablet_replica.cc:330] stopping tablet replica
I20260929 19:03:53.851133 16511 raft_consensus.cc:2229] T c7782e56046a47539036f874ace86f91 P 45494feaa6df4f32807cb11e2b10a536 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 19:03:53.851194 16511 raft_consensus.cc:2258] T c7782e56046a47539036f874ace86f91 P 45494feaa6df4f32807cb11e2b10a536 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 19:03:53.851475 16511 ts_tablet_manager.cc:1905] T c7782e56046a47539036f874ace86f91 P 45494feaa6df4f32807cb11e2b10a536: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 19:03:53.853749 16511 ts_tablet_manager.cc:1918] T c7782e56046a47539036f874ace86f91 P 45494feaa6df4f32807cb11e2b10a536: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 19:03:53.853816 16511 log.cc:1196] T c7782e56046a47539036f874ace86f91 P 45494feaa6df4f32807cb11e2b10a536: Deleting WAL directory at /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/minicluster-data/ts-0/wal/wals/c7782e56046a47539036f874ace86f91
I20260929 19:03:53.854027 16511 ts_tablet_manager.cc:1939] T c7782e56046a47539036f874ace86f91 P 45494feaa6df4f32807cb11e2b10a536: Deleting consensus metadata
I20260929 19:03:53.854518 16299 catalog_manager.cc:4841] TS 45494feaa6df4f32807cb11e2b10a536 (127.0.98.1:42445): tablet c7782e56046a47539036f874ace86f91 (table test-table [id=749c6d60b5ee421f867df493103e1a5b]) successfully deleted
W20260929 19:03:53.903735 16506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:03:53.903884 16506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:03:53.903903 16506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:03:53.904872 16506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:03:53.904902 16506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:03:53.904925 16506 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 19:03:53.906126 16506 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:35361
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=37303
--tserver_master_addrs=127.0.98.62:40647
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 18:09:13 UTC on ccycloud.quasar-hpkalf.root.comops.site
build id 4680913
I20260929 19:03:53.906273 16506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:03:53.906410 16506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:03:53.910987 16517 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:03:53.910989 16516 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:03:53.911016 16519 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:03:53.911154 16506 server_base.cc:1034] running on GCE node
I20260929 19:03:53.911374 16506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:03:53.911629 16506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:03:53.912794 16506 hybrid_clock.cc:648] HybridClock initialized: now 1790708633912794 us; error 37 us; skew 500 ppm
Sep 29 19:03:53 dist-test-slave-lqkv krb5kdc[15736](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708633, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:03:53.915119 16506 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 19:03:53.915500 16506 webserver.cc:460] Webserver started at http://127.0.98.2:37303/ using document root <none> and password file <none>
I20260929 19:03:53.915710 16506 fs_manager.cc:359] Metadata directory not provided
I20260929 19:03:53.915783 16506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:03:53.916915 16506 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 19:03:53.917536 16526 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:53.917727 16506 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 19:03:53.917814 16506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/minicluster-data/ts-1/wal
uuid: "c8150c87f2684d4c937a086a3ef65e2a"
format_stamp: "Formatted at 2026-09-29 19:03:33 on dist-test-slave-lqkv"
server_key: "e7b0ad82b16a7d630e34d8eebaf9958f"
server_key_iv: "076a095a7e7f74a0625173fc5676be78"
server_key_version: "encryptionkey@0"
I20260929 19:03:53.918114 16506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:03:53.948187 16506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:03:53.949913 16506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:03:53.950066 16506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:03:53.950278 16506 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:03:53.950554 16506 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:03:53.950606 16506 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:53.950644 16506 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:03:53.950671 16506 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:53.956497 16506 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35361
I20260929 19:03:53.956530 16639 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35361 every 8 connection(s)
I20260929 19:03:53.956852 16506 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/minicluster-data/ts-1/data/info.pb
I20260929 19:03:53.957652 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUbnLHE/build/release/bin/kudu as pid 16506
I20260929 19:03:53.957929 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUbnLHE/build/release/bin/kudu
/tmp/dist-test-taskUbnLHE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:44775
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:40647
--webserver_port=43749
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 19:03:53 dist-test-slave-lqkv krb5kdc[15736](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708633, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 19:03:53.966773 16505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:53.957463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45159 (local address 127.0.98.62:40647)
0929 19:03:53.957501 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:53.957503 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:53.957704 (+ 201us) server_negotiation.cc:407] Connection header received
0929 19:03:53.958099 (+ 395us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:53.958101 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:53.958116 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:53.958174 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:53.959377 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:53.960129 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:53.961116 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:53.961461 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:53.961539 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:53.963933 (+ 2394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:03:53.963934 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:03:53.963934 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:03:53.963940 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:03:53.965351 (+ 1411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:53.965819 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:53.965820 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:53.965820 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:53.965865 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:53.966174 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:53.966174 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:53.966174 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:53.966285 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:03:53.966328 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:53.966648 (+ 320us) server_negotiation.cc:299] Negotiation successful
0929 19:03:53.966713 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260929 19:03:53.967080 16642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:53.957429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40647 (local address 127.0.98.2:45159)
0929 19:03:53.957573 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:53.957618 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:53.958005 (+ 387us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:53.958303 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:53.958308 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:53.959012 (+ 704us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:03:53.959281 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:53.959286 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:53.960222 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:53.960222 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:53.961032 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:53.961035 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:53.961542 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:53.961543 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:53.961636 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:53.962102 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:03:53.962109 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:03:53.963848 (+ 1739us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:03:53.965427 (+ 1579us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:03:53.965434 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:03:53.965436 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:03:53.965740 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:03:53.965923 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:03:53.965924 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:03:53.965924 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:03:53.966078 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:03:53.966347 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:03:53.966354 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:03:53.966512 (+ 158us) client_negotiation.cc:769] Sending connection context
0929 19:03:53.966634 (+ 122us) client_negotiation.cc:240] Negotiation successful
0929 19:03:53.966798 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":81,"threads_started":1}
I20260929 19:03:53.967634 16640 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40647
I20260929 19:03:53.967741 16640 heartbeater.cc:464] Registering TS with master...
I20260929 19:03:53.967913 16640 heartbeater.cc:511] Master 127.0.98.62:40647 requested a full tablet report, sending...
I20260929 19:03:53.968350 16314 ts_manager.cc:194] Registered new tserver with Master: c8150c87f2684d4c937a086a3ef65e2a (127.0.98.2:35361)
I20260929 19:03:53.968883 16314 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:45159
W20260929 19:03:54.034612 16643 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:03:54.034770 16643 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:03:54.034789 16643 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:03:54.035815 16643 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:03:54.035846 16643 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:03:54.035871 16643 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 19:03:54.037184 16643 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:44775
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=43749
--tserver_master_addrs=127.0.98.62:40647
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16643
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 18:09:13 UTC on ccycloud.quasar-hpkalf.root.comops.site
build id 4680913
I20260929 19:03:54.037351 16643 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:03:54.037492 16643 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:03:54.041901 16651 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:03:54.041901 16648 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:03:54.041903 16649 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:03:54.041916 16643 server_base.cc:1034] running on GCE node
I20260929 19:03:54.042315 16643 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:03:54.042541 16643 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:03:54.043708 16643 hybrid_clock.cc:648] HybridClock initialized: now 1790708634043699 us; error 26 us; skew 500 ppm
Sep 29 19:03:54 dist-test-slave-lqkv krb5kdc[15736](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708634, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:03:54.045972 16643 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 19:03:54.046350 16643 webserver.cc:460] Webserver started at http://127.0.98.3:43749/ using document root <none> and password file <none>
I20260929 19:03:54.046553 16643 fs_manager.cc:359] Metadata directory not provided
I20260929 19:03:54.046629 16643 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:03:54.047852 16643 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 19:03:54.048359 16658 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:54.048492 16643 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 19:03:54.048575 16643 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/minicluster-data/ts-2/wal
uuid: "43c7b519591f4119a105016fef7c4656"
format_stamp: "Formatted at 2026-09-29 19:03:33 on dist-test-slave-lqkv"
server_key: "9dbbaaf51b9418e023cc82e84ca5e550"
server_key_iv: "c339eb06ae9f36bf9a77ec5eb2974423"
server_key_version: "encryptionkey@0"
I20260929 19:03:54.048900 16643 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:03:54.067243 16643 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:03:54.068975 16643 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:03:54.069124 16643 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:03:54.069325 16643 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:03:54.069600 16643 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:03:54.069658 16643 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:54.069705 16643 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:03:54.069736 16643 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:54.075332 16643 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44775
I20260929 19:03:54.075372 16771 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44775 every 8 connection(s)
I20260929 19:03:54.075651 16643 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/minicluster-data/ts-2/data/info.pb
Sep 29 19:03:54 dist-test-slave-lqkv krb5kdc[15736](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708634, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 19:03:54.082907 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUbnLHE/build/release/bin/kudu as pid 16643
I20260929 19:03:54.083707 16505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:54.076217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56945 (local address 127.0.98.62:40647)
0929 19:03:54.076277 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:54.076279 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:54.076481 (+ 202us) server_negotiation.cc:407] Connection header received
0929 19:03:54.076946 (+ 465us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:54.076947 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:54.076956 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:54.077005 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:54.077767 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:54.078210 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:54.079088 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:54.079258 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:54.079305 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:54.081156 (+ 1851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:03:54.081158 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:03:54.081158 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:03:54.081165 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:03:54.082328 (+ 1163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:54.082700 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:54.082701 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:54.082702 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:54.082761 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:54.082995 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:54.082996 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:54.082996 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:54.083123 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:03:54.083192 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:54.083517 (+ 325us) server_negotiation.cc:299] Negotiation successful
0929 19:03:54.083614 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260929 19:03:54.083854 16774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:54.076213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40647 (local address 127.0.98.3:56945)
0929 19:03:54.076360 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:54.076396 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:54.076862 (+ 466us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:54.077050 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:54.077053 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:54.077538 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:03:54.077701 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:54.077705 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:54.078284 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:54.078285 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:54.079023 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:54.079025 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:54.079323 (+ 298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:54.079324 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:54.079408 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:54.079802 (+ 394us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:03:54.079809 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:03:54.081066 (+ 1257us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:03:54.082408 (+ 1342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:03:54.082413 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:03:54.082415 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:03:54.082627 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:03:54.082832 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:03:54.082833 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:03:54.082834 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:03:54.082931 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:03:54.083197 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:03:54.083202 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:03:54.083359 (+ 157us) client_negotiation.cc:769] Sending connection context
0929 19:03:54.083482 (+ 123us) client_negotiation.cc:240] Negotiation successful
0929 19:03:54.083633 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":82,"threads_started":1}
I20260929 19:03:54.084286 16772 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40647
I20260929 19:03:54.084380 16772 heartbeater.cc:464] Registering TS with master...
I20260929 19:03:54.084569 16772 heartbeater.cc:511] Master 127.0.98.62:40647 requested a full tablet report, sending...
I20260929 19:03:54.084935 16314 ts_manager.cc:194] Registered new tserver with Master: 43c7b519591f4119a105016fef7c4656 (127.0.98.3:44775)
I20260929 19:03:54.085366 16314 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:56945
I20260929 19:03:54.089427 16775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:54.083317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52344 (local address 127.0.98.62:40647)
0929 19:03:54.083417 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:54.083419 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:54.083573 (+ 154us) server_negotiation.cc:407] Connection header received
0929 19:03:54.083649 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:54.083650 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:54.083662 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:54.083740 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:54.084486 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:54.085134 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:54.086050 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:54.086228 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:54.086282 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:54.087396 (+ 1114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:03:54.087396 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:03:54.087397 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:03:54.087401 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:03:54.088510 (+ 1109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:54.088799 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:54.088799 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:54.088831 (+ 32us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:54.088886 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:54.089044 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:54.089045 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:54.089045 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:54.089137 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:03:54.089239 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:54.089293 (+ 54us) server_negotiation.cc:299] Negotiation successful
0929 19:03:54.089348 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82,"thread_start_us":59,"threads_started":1}
I20260929 19:03:54.090139 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 19:03:54.092324 16775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:54.090449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52352 (local address 127.0.98.62:40647)
0929 19:03:54.090481 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:54.090482 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:54.090559 (+ 77us) server_negotiation.cc:407] Connection header received
0929 19:03:54.090603 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:54.090604 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:54.090628 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:54.090684 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 19:03:54.090753 (+ 69us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:54.091174 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:54.091827 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:54.091999 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:54.092046 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:54.092160 (+ 114us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 19:03:54.092193 (+ 33us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0929 19:03:54.092287 (+ 94us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:52352: token expired
Metrics: {"server-negotiator.queue_time_us":16}
W20260929 19:03:54.092387 16775 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:52352: token expired
W20260929 19:03:54.092483 16246 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.98.62:40647)
I20260929 19:03:54.097481 16775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:54.092734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52358 (local address 127.0.98.62:40647)
0929 19:03:54.092764 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:54.092766 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:54.092773 (+ 7us) server_negotiation.cc:407] Connection header received
0929 19:03:54.092795 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:54.092796 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:54.092830 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:54.092882 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:03:54.093288 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:54.093669 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:54.094263 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:54.094415 (+ 152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:54.094457 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:54.095353 (+ 896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:03:54.095354 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:03:54.095354 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:03:54.095360 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:03:54.096656 (+ 1296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:54.096986 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:54.096987 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:54.096987 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:54.097033 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:03:54.097154 (+ 121us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:03:54.097155 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:03:54.097155 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:03:54.097247 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:03:54.097308 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:54.097381 (+ 73us) server_negotiation.cc:299] Negotiation successful
0929 19:03:54.097433 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260929 19:03:54.098397 16314 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52358:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 19:03:54.099906 16314 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 19:03:54.101804 16418 tablet_service.cc:1467] Processing CreateTablet for tablet 34be0995612d4d4bb8103b81d56677e0 (DEFAULT_TABLE table=test-table [id=072c803cb25446afa9cff07df33431ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:03:54.101986 16418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34be0995612d4d4bb8103b81d56677e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:03:54.103576 16394 tablet_bootstrap.cc:492] T 34be0995612d4d4bb8103b81d56677e0 P 45494feaa6df4f32807cb11e2b10a536: Bootstrap starting.
I20260929 19:03:54.103870 16394 tablet_bootstrap.cc:654] T 34be0995612d4d4bb8103b81d56677e0 P 45494feaa6df4f32807cb11e2b10a536: Neither blocks nor log segments found. Creating new log.
I20260929 19:03:54.104362 16394 tablet_bootstrap.cc:492] T 34be0995612d4d4bb8103b81d56677e0 P 45494feaa6df4f32807cb11e2b10a536: No bootstrap required, opened a new log
I20260929 19:03:54.104410 16394 ts_tablet_manager.cc:1397] T 34be0995612d4d4bb8103b81d56677e0 P 45494feaa6df4f32807cb11e2b10a536: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260929 19:03:54.104512 16394 raft_consensus.cc:348] T 34be0995612d4d4bb8103b81d56677e0 P 45494feaa6df4f32807cb11e2b10a536 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43c7b519591f4119a105016fef7c4656" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44775 } } peers { permanent_uuid: "c8150c87f2684d4c937a086a3ef65e2a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35361 } } peers { permanent_uuid: "45494feaa6df4f32807cb11e2b10a536" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42445 } }
I20260929 19:03:54.104600 16394 raft_consensus.cc:374] T 34be0995612d4d4bb8103b81d56677e0 P 45494feaa6df4f32807cb11e2b10a536 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:03:54.104646 16394 raft_consensus.cc:729] T 34be0995612d4d4bb8103b81d56677e0 P 45494feaa6df4f32807cb11e2b10a536 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45494feaa6df4f32807cb11e2b10a536, State: Initialized, Role: FOLLOWER
I20260929 19:03:54.104686 16394 consensus_queue.cc:260] T 34be0995612d4d4bb8103b81d56677e0 P 45494feaa6df4f32807cb11e2b10a536 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43c7b519591f4119a105016fef7c4656" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44775 } } peers { permanent_uuid: "c8150c87f2684d4c937a086a3ef65e2a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35361 } } peers { permanent_uuid: "45494feaa6df4f32807cb11e2b10a536" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42445 } }
I20260929 19:03:54.104771 16394 ts_tablet_manager.cc:1428] T 34be0995612d4d4bb8103b81d56677e0 P 45494feaa6df4f32807cb11e2b10a536: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260929 19:03:54.105302 16779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:54.101790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53018 (local address 127.0.98.3:44775)
0929 19:03:54.101963 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:54.101968 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:54.101979 (+ 11us) server_negotiation.cc:407] Connection header received
0929 19:03:54.102017 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:54.102019 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:54.102036 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:54.102148 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:03:54.102235 (+ 87us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:54.103097 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:54.104017 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:54.104933 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:54.105042 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:54.105111 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:54.105199 (+ 88us) server_negotiation.cc:299] Negotiation successful
0929 19:03:54.105233 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":114,"threads_started":1}
I20260929 19:03:54.105304 16509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:54.101596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44775 (local address 127.0.0.1:53018)
0929 19:03:54.101628 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:54.101637 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:54.101668 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:54.102144 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:54.102144 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:54.102145 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:03:54.102181 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:54.102182 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:54.103172 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:54.103173 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:54.103938 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:54.103940 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:54.105046 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:54.105047 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:54.105119 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:54.105122 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 19:03:54.105181 (+ 59us) client_negotiation.cc:240] Negotiation successful
0929 19:03:54.105224 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16,"mutex_wait_us":5}
I20260929 19:03:54.105964 16778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:54.101688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35361 (local address 127.0.0.1:33688)
0929 19:03:54.102016 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:54.102026 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:54.102075 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:54.102268 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:54.102268 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:54.102270 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:03:54.102300 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:54.102301 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:54.103621 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:54.103622 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:54.104897 (+ 1275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:54.104900 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:54.105805 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:54.105805 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:54.105860 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:54.105863 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 19:03:54.105902 (+ 39us) client_negotiation.cc:240] Negotiation successful
0929 19:03:54.105924 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":80,"threads_started":1}
I20260929 19:03:54.106036 16780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:54.101795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33688 (local address 127.0.98.2:35361)
0929 19:03:54.102120 (+ 325us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:54.102124 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:54.102135 (+ 11us) server_negotiation.cc:407] Connection header received
0929 19:03:54.102173 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:54.102175 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:54.102190 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:54.102274 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:03:54.102357 (+ 83us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:54.103513 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:54.104978 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:54.105728 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:54.105797 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:54.105857 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:54.105923 (+ 66us) server_negotiation.cc:299] Negotiation successful
0929 19:03:54.105958 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":116,"threads_started":1}
I20260929 19:03:54.105983 16706 tablet_service.cc:1467] Processing CreateTablet for tablet 34be0995612d4d4bb8103b81d56677e0 (DEFAULT_TABLE table=test-table [id=072c803cb25446afa9cff07df33431ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:03:54.106295 16706 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34be0995612d4d4bb8103b81d56677e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:03:54.106606 16574 tablet_service.cc:1467] Processing CreateTablet for tablet 34be0995612d4d4bb8103b81d56677e0 (DEFAULT_TABLE table=test-table [id=072c803cb25446afa9cff07df33431ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:03:54.106904 16574 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34be0995612d4d4bb8103b81d56677e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:03:54.108785 16782 tablet_bootstrap.cc:492] T 34be0995612d4d4bb8103b81d56677e0 P 43c7b519591f4119a105016fef7c4656: Bootstrap starting.
I20260929 19:03:54.109364 16782 tablet_bootstrap.cc:654] T 34be0995612d4d4bb8103b81d56677e0 P 43c7b519591f4119a105016fef7c4656: Neither blocks nor log segments found. Creating new log.
I20260929 19:03:54.109555 16783 tablet_bootstrap.cc:492] T 34be0995612d4d4bb8103b81d56677e0 P c8150c87f2684d4c937a086a3ef65e2a: Bootstrap starting.
I20260929 19:03:54.109930 16782 log.cc:824] T 34be0995612d4d4bb8103b81d56677e0 P 43c7b519591f4119a105016fef7c4656: Log is configured to *not* fsync() on all Append() calls
I20260929 19:03:54.110140 16783 tablet_bootstrap.cc:654] T 34be0995612d4d4bb8103b81d56677e0 P c8150c87f2684d4c937a086a3ef65e2a: Neither blocks nor log segments found. Creating new log.
I20260929 19:03:54.110603 16782 tablet_bootstrap.cc:492] T 34be0995612d4d4bb8103b81d56677e0 P 43c7b519591f4119a105016fef7c4656: No bootstrap required, opened a new log
I20260929 19:03:54.110658 16783 log.cc:824] T 34be0995612d4d4bb8103b81d56677e0 P c8150c87f2684d4c937a086a3ef65e2a: Log is configured to *not* fsync() on all Append() calls
I20260929 19:03:54.110689 16782 ts_tablet_manager.cc:1397] T 34be0995612d4d4bb8103b81d56677e0 P 43c7b519591f4119a105016fef7c4656: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260929 19:03:54.111317 16783 tablet_bootstrap.cc:492] T 34be0995612d4d4bb8103b81d56677e0 P c8150c87f2684d4c937a086a3ef65e2a: No bootstrap required, opened a new log
I20260929 19:03:54.111392 16783 ts_tablet_manager.cc:1397] T 34be0995612d4d4bb8103b81d56677e0 P c8150c87f2684d4c937a086a3ef65e2a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 19:03:54.112461 16782 raft_consensus.cc:348] T 34be0995612d4d4bb8103b81d56677e0 P 43c7b519591f4119a105016fef7c4656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43c7b519591f4119a105016fef7c4656" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44775 } } peers { permanent_uuid: "c8150c87f2684d4c937a086a3ef65e2a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35361 } } peers { permanent_uuid: "45494feaa6df4f32807cb11e2b10a536" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42445 } }
I20260929 19:03:54.112586 16782 raft_consensus.cc:374] T 34be0995612d4d4bb8103b81d56677e0 P 43c7b519591f4119a105016fef7c4656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:03:54.112607 16782 raft_consensus.cc:729] T 34be0995612d4d4bb8103b81d56677e0 P 43c7b519591f4119a105016fef7c4656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43c7b519591f4119a105016fef7c4656, State: Initialized, Role: FOLLOWER
I20260929 19:03:54.112727 16782 consensus_queue.cc:260] T 34be0995612d4d4bb8103b81d56677e0 P 43c7b519591f4119a105016fef7c4656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43c7b519591f4119a105016fef7c4656" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44775 } } peers { permanent_uuid: "c8150c87f2684d4c937a086a3ef65e2a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35361 } } peers { permanent_uuid: "45494feaa6df4f32807cb11e2b10a536" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42445 } }
I20260929 19:03:54.113045 16772 heartbeater.cc:503] Master 127.0.98.62:40647 was elected leader, sending a full tablet report...
I20260929 19:03:54.113030 16782 ts_tablet_manager.cc:1428] T 34be0995612d4d4bb8103b81d56677e0 P 43c7b519591f4119a105016fef7c4656: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260929 19:03:54.113284 16783 raft_consensus.cc:348] T 34be0995612d4d4bb8103b81d56677e0 P c8150c87f2684d4c937a086a3ef65e2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43c7b519591f4119a105016fef7c4656" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44775 } } peers { permanent_uuid: "c8150c87f2684d4c937a086a3ef65e2a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35361 } } peers { permanent_uuid: "45494feaa6df4f32807cb11e2b10a536" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42445 } }
I20260929 19:03:54.113461 16783 raft_consensus.cc:374] T 34be0995612d4d4bb8103b81d56677e0 P c8150c87f2684d4c937a086a3ef65e2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:03:54.113499 16783 raft_consensus.cc:729] T 34be0995612d4d4bb8103b81d56677e0 P c8150c87f2684d4c937a086a3ef65e2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8150c87f2684d4c937a086a3ef65e2a, State: Initialized, Role: FOLLOWER
I20260929 19:03:54.113616 16783 consensus_queue.cc:260] T 34be0995612d4d4bb8103b81d56677e0 P c8150c87f2684d4c937a086a3ef65e2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43c7b519591f4119a105016fef7c4656" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44775 } } peers { permanent_uuid: "c8150c87f2684d4c937a086a3ef65e2a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35361 } } peers { permanent_uuid: "45494feaa6df4f32807cb11e2b10a536" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42445 } }
I20260929 19:03:54.113869 16783 ts_tablet_manager.cc:1428] T 34be0995612d4d4bb8103b81d56677e0 P c8150c87f2684d4c937a086a3ef65e2a: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260929 19:03:54.113911 16640 heartbeater.cc:503] Master 127.0.98.62:40647 was elected leader, sending a full tablet report...
I20260929 19:03:54.126080 16787 raft_consensus.cc:482] T 34be0995612d4d4bb8103b81d56677e0 P c8150c87f2684d4c937a086a3ef65e2a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 19:03:54.126144 16787 raft_consensus.cc:504] T 34be0995612d4d4bb8103b81d56677e0 P c8150c87f2684d4c937a086a3ef65e2a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43c7b519591f4119a105016fef7c4656" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44775 } } peers { permanent_uuid: "c8150c87f2684d4c937a086a3ef65e2a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35361 } } peers { permanent_uuid: "45494feaa6df4f32807cb11e2b10a536" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42445 } }
I20260929 19:03:54.126405 16787 leader_election.cc:288] T 34be0995612d4d4bb8103b81d56677e0 P c8150c87f2684d4c937a086a3ef65e2a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 43c7b519591f4119a105016fef7c4656 (127.0.98.3:44775), 45494feaa6df4f32807cb11e2b10a536 (127.0.98.1:42445)
I20260929 19:03:54.129925 16510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:54.126585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33397 (local address 127.0.98.1:42445)
0929 19:03:54.126626 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:54.126628 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:54.126765 (+ 137us) server_negotiation.cc:407] Connection header received
0929 19:03:54.126793 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:54.126794 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:54.126806 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:54.126861 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:03:54.126964 (+ 103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:54.127668 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:54.129103 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:54.129616 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:54.129691 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:54.129700 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:54.129838 (+ 138us) server_negotiation.cc:299] Negotiation successful
0929 19:03:54.129873 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260929 19:03:54.129920 16779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:54.126596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50399 (local address 127.0.98.3:44775)
0929 19:03:54.126630 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:54.126632 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:54.126658 (+ 26us) server_negotiation.cc:407] Connection header received
0929 19:03:54.126685 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:54.126686 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:54.126693 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:54.126730 (+ 37us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:03:54.126882 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:54.127642 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:54.129103 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:54.129618 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:54.129684 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:54.129692 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:54.129836 (+ 144us) server_negotiation.cc:299] Negotiation successful
0929 19:03:54.129868 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260929 19:03:54.129932 16642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:54.126518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44775 (local address 127.0.98.2:50399)
0929 19:03:54.126562 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:54.126574 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:54.126625 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:54.126752 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:54.126753 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:54.126755 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:03:54.126816 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:54.126817 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:54.127738 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:54.127738 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:54.129020 (+ 1282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:54.129023 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:54.129695 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:54.129695 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:54.129766 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:54.129767 (+ 1us) client_negotiation.cc:769] Sending connection context
0929 19:03:54.129817 (+ 50us) client_negotiation.cc:240] Negotiation successful
0929 19:03:54.129855 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24,"mutex_wait_us":62}
I20260929 19:03:54.129938 16788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:54.126536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42445 (local address 127.0.98.2:33397)
0929 19:03:54.126676 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:54.126691 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 19:03:54.126752 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:03:54.126847 (+ 95us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:03:54.126848 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:03:54.126849 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:03:54.126908 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:54.126909 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:54.127742 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:54.127743 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:54.129020 (+ 1277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:03:54.129023 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:03:54.129695 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:03:54.129695 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:03:54.129762 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:54.129766 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 19:03:54.129817 (+ 51us) client_negotiation.cc:240] Negotiation successful
0929 19:03:54.129855 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"mutex_wait_us":63,"thread_start_us":84,"threads_started":1}
I20260929 19:03:54.130283 16726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34be0995612d4d4bb8103b81d56677e0" candidate_uuid: "c8150c87f2684d4c937a086a3ef65e2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43c7b519591f4119a105016fef7c4656" is_pre_election: true
I20260929 19:03:54.130414 16726 raft_consensus.cc:2454] T 34be0995612d4d4bb8103b81d56677e0 P 43c7b519591f4119a105016fef7c4656 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8150c87f2684d4c937a086a3ef65e2a in term 0.
I20260929 19:03:54.130465 16456 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34be0995612d4d4bb8103b81d56677e0" candidate_uuid: "c8150c87f2684d4c937a086a3ef65e2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45494feaa6df4f32807cb11e2b10a536" is_pre_election: true
I20260929 19:03:54.130633 16456 raft_consensus.cc:2454] T 34be0995612d4d4bb8103b81d56677e0 P 45494feaa6df4f32807cb11e2b10a536 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8150c87f2684d4c937a086a3ef65e2a in term 0.
I20260929 19:03:54.130636 16529 leader_election.cc:302] T 34be0995612d4d4bb8103b81d56677e0 P c8150c87f2684d4c937a086a3ef65e2a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43c7b519591f4119a105016fef7c4656, c8150c87f2684d4c937a086a3ef65e2a; no voters:
I20260929 19:03:54.130739 16787 raft_consensus.cc:2781] T 34be0995612d4d4bb8103b81d56677e0 P c8150c87f2684d4c937a086a3ef65e2a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 19:03:54.130795 16787 raft_consensus.cc:482] T 34be0995612d4d4bb8103b81d56677e0 P c8150c87f2684d4c937a086a3ef65e2a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 19:03:54.130829 16787 raft_consensus.cc:3037] T 34be0995612d4d4bb8103b81d56677e0 P c8150c87f2684d4c937a086a3ef65e2a [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:03:54.131413 16787 raft_consensus.cc:504] T 34be0995612d4d4bb8103b81d56677e0 P c8150c87f2684d4c937a086a3ef65e2a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43c7b519591f4119a105016fef7c4656" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44775 } } peers { permanent_uuid: "c8150c87f2684d4c937a086a3ef65e2a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35361 } } peers { permanent_uuid: "45494feaa6df4f32807cb11e2b10a536" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42445 } }
I20260929 19:03:54.131539 16787 leader_election.cc:288] T 34be0995612d4d4bb8103b81d56677e0 P c8150c87f2684d4c937a086a3ef65e2a [CANDIDATE]: Term 1 election: Requested vote from peers 43c7b519591f4119a105016fef7c4656 (127.0.98.3:44775), 45494feaa6df4f32807cb11e2b10a536 (127.0.98.1:42445)
I20260929 19:03:54.131681 16456 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34be0995612d4d4bb8103b81d56677e0" candidate_uuid: "c8150c87f2684d4c937a086a3ef65e2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45494feaa6df4f32807cb11e2b10a536"
I20260929 19:03:54.131737 16726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34be0995612d4d4bb8103b81d56677e0" candidate_uuid: "c8150c87f2684d4c937a086a3ef65e2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43c7b519591f4119a105016fef7c4656"
I20260929 19:03:54.131772 16456 raft_consensus.cc:3037] T 34be0995612d4d4bb8103b81d56677e0 P 45494feaa6df4f32807cb11e2b10a536 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:03:54.131824 16726 raft_consensus.cc:3037] T 34be0995612d4d4bb8103b81d56677e0 P 43c7b519591f4119a105016fef7c4656 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:03:54.132431 16726 raft_consensus.cc:2454] T 34be0995612d4d4bb8103b81d56677e0 P 43c7b519591f4119a105016fef7c4656 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8150c87f2684d4c937a086a3ef65e2a in term 1.
I20260929 19:03:54.132599 16529 leader_election.cc:302] T 34be0995612d4d4bb8103b81d56677e0 P c8150c87f2684d4c937a086a3ef65e2a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43c7b519591f4119a105016fef7c4656, c8150c87f2684d4c937a086a3ef65e2a; no voters:
I20260929 19:03:54.132694 16787 raft_consensus.cc:2781] T 34be0995612d4d4bb8103b81d56677e0 P c8150c87f2684d4c937a086a3ef65e2a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:03:54.132879 16787 raft_consensus.cc:686] T 34be0995612d4d4bb8103b81d56677e0 P c8150c87f2684d4c937a086a3ef65e2a [term 1 LEADER]: Becoming Leader. State: Replica: c8150c87f2684d4c937a086a3ef65e2a, State: Running, Role: LEADER
I20260929 19:03:54.132961 16787 consensus_queue.cc:237] T 34be0995612d4d4bb8103b81d56677e0 P c8150c87f2684d4c937a086a3ef65e2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43c7b519591f4119a105016fef7c4656" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44775 } } peers { permanent_uuid: "c8150c87f2684d4c937a086a3ef65e2a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35361 } } peers { permanent_uuid: "45494feaa6df4f32807cb11e2b10a536" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42445 } }
I20260929 19:03:54.133016 16456 raft_consensus.cc:2454] T 34be0995612d4d4bb8103b81d56677e0 P 45494feaa6df4f32807cb11e2b10a536 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8150c87f2684d4c937a086a3ef65e2a in term 1.
I20260929 19:03:54.133695 16314 catalog_manager.cc:5495] T 34be0995612d4d4bb8103b81d56677e0 P c8150c87f2684d4c937a086a3ef65e2a reported cstate change: term changed from 0 to 1, leader changed from <none> to c8150c87f2684d4c937a086a3ef65e2a (127.0.98.2). New cstate: current_term: 1 leader_uuid: "c8150c87f2684d4c937a086a3ef65e2a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43c7b519591f4119a105016fef7c4656" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8150c87f2684d4c937a086a3ef65e2a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "45494feaa6df4f32807cb11e2b10a536" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42445 } health_report { overall_health: UNKNOWN } } }
I20260929 19:03:54.142158 16780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:03:54.139899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33690 (local address 127.0.98.2:35361)
0929 19:03:54.139937 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0929 19:03:54.139939 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:03:54.139947 (+ 8us) server_negotiation.cc:407] Connection header received
0929 19:03:54.140004 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:03:54.140005 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:03:54.140019 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:03:54.140079 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 19:03:54.140199 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:54.140588 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:54.141385 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:03:54.141579 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:03:54.141658 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:03:54.141812 (+ 154us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 19:03:54.141897 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 19:03:54.142005 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0929 19:03:54.142054 (+ 49us) server_negotiation.cc:299] Negotiation successful
0929 19:03:54.142071 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260929 19:03:54.143488 16456 raft_consensus.cc:1261] T 34be0995612d4d4bb8103b81d56677e0 P 45494feaa6df4f32807cb11e2b10a536 [term 1 FOLLOWER]: Refusing update from remote peer c8150c87f2684d4c937a086a3ef65e2a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 19:03:54.143488 16726 raft_consensus.cc:1261] T 34be0995612d4d4bb8103b81d56677e0 P 43c7b519591f4119a105016fef7c4656 [term 1 FOLLOWER]: Refusing update from remote peer c8150c87f2684d4c937a086a3ef65e2a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 19:03:54.143810 16787 consensus_queue.cc:1035] T 34be0995612d4d4bb8103b81d56677e0 P c8150c87f2684d4c937a086a3ef65e2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "45494feaa6df4f32807cb11e2b10a536" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 19:03:54.143916 16789 consensus_queue.cc:1035] T 34be0995612d4d4bb8103b81d56677e0 P c8150c87f2684d4c937a086a3ef65e2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "43c7b519591f4119a105016fef7c4656" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 19:03:54.145586 16791 mvcc.cc:205] Tried to move back new op lower bound from 7334742565449777152 to 7334742565409316864. Current Snapshot: MvccSnapshot[applied={T|T < 7334742565449777152}]
I20260929 19:03:54.147969 16314 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52358:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 19:03:54.148051 16314 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52358:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 19:03:54.148689 16314 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 37a91a7695be4407b44d568eae618e7b: Sending DeleteTablet for 3 replicas of tablet 34be0995612d4d4bb8103b81d56677e0
I20260929 19:03:54.148921 16418 tablet_service.cc:1514] Processing DeleteTablet for tablet 34be0995612d4d4bb8103b81d56677e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:03:54 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37420
I20260929 19:03:54.148988 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUbnLHE/build/release/bin/kudu with pid 16352
I20260929 19:03:54.148974 16574 tablet_service.cc:1514] Processing DeleteTablet for tablet 34be0995612d4d4bb8103b81d56677e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:03:54 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33688
I20260929 19:03:54.149124 16511 tablet_replica.cc:330] T 34be0995612d4d4bb8103b81d56677e0 P 45494feaa6df4f32807cb11e2b10a536: stopping tablet replica
I20260929 19:03:54.149173 16511 raft_consensus.cc:2229] T 34be0995612d4d4bb8103b81d56677e0 P 45494feaa6df4f32807cb11e2b10a536 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 19:03:54.149230 16511 raft_consensus.cc:2258] T 34be0995612d4d4bb8103b81d56677e0 P 45494feaa6df4f32807cb11e2b10a536 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 19:03:54.149501 16706 tablet_service.cc:1514] Processing DeleteTablet for tablet 34be0995612d4d4bb8103b81d56677e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:03:54 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53018
I20260929 19:03:54.149668 16801 tablet_replica.cc:330] T 34be0995612d4d4bb8103b81d56677e0 P 43c7b519591f4119a105016fef7c4656: stopping tablet replica
I20260929 19:03:54.149781 16801 raft_consensus.cc:2229] T 34be0995612d4d4bb8103b81d56677e0 P 43c7b519591f4119a105016fef7c4656 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 19:03:54.149892 16801 raft_consensus.cc:2258] T 34be0995612d4d4bb8103b81d56677e0 P 43c7b519591f4119a105016fef7c4656 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 19:03:54.150285 16801 ts_tablet_manager.cc:1905] T 34be0995612d4d4bb8103b81d56677e0 P 43c7b519591f4119a105016fef7c4656: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 19:03:54.152038 16801 ts_tablet_manager.cc:1918] T 34be0995612d4d4bb8103b81d56677e0 P 43c7b519591f4119a105016fef7c4656: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 19:03:54.152153 16801 log.cc:1196] T 34be0995612d4d4bb8103b81d56677e0 P 43c7b519591f4119a105016fef7c4656: Deleting WAL directory at /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/minicluster-data/ts-2/wal/wals/34be0995612d4d4bb8103b81d56677e0
I20260929 19:03:54.152488 16801 ts_tablet_manager.cc:1939] T 34be0995612d4d4bb8103b81d56677e0 P 43c7b519591f4119a105016fef7c4656: Deleting consensus metadata
I20260929 19:03:54.152952 16300 catalog_manager.cc:4841] TS 43c7b519591f4119a105016fef7c4656 (127.0.98.3:44775): tablet 34be0995612d4d4bb8103b81d56677e0 (table test-table [id=072c803cb25446afa9cff07df33431ea]) successfully deleted
I20260929 19:03:54.153090 16800 tablet_replica.cc:330] T 34be0995612d4d4bb8103b81d56677e0 P c8150c87f2684d4c937a086a3ef65e2a: stopping tablet replica
I20260929 19:03:54.153209 16800 raft_consensus.cc:2229] T 34be0995612d4d4bb8103b81d56677e0 P c8150c87f2684d4c937a086a3ef65e2a [term 1 LEADER]: Raft consensus shutting down.
I20260929 19:03:54.153302 16800 raft_consensus.cc:2258] T 34be0995612d4d4bb8103b81d56677e0 P c8150c87f2684d4c937a086a3ef65e2a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 19:03:54.153626 16800 ts_tablet_manager.cc:1905] T 34be0995612d4d4bb8103b81d56677e0 P c8150c87f2684d4c937a086a3ef65e2a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 19:03:54.154951 16800 ts_tablet_manager.cc:1918] T 34be0995612d4d4bb8103b81d56677e0 P c8150c87f2684d4c937a086a3ef65e2a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 19:03:54.155015 16800 log.cc:1196] T 34be0995612d4d4bb8103b81d56677e0 P c8150c87f2684d4c937a086a3ef65e2a: Deleting WAL directory at /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/minicluster-data/ts-1/wal/wals/34be0995612d4d4bb8103b81d56677e0
I20260929 19:03:54.155262 16800 ts_tablet_manager.cc:1939] T 34be0995612d4d4bb8103b81d56677e0 P c8150c87f2684d4c937a086a3ef65e2a: Deleting consensus metadata
I20260929 19:03:54.155601 16300 catalog_manager.cc:4841] TS c8150c87f2684d4c937a086a3ef65e2a (127.0.98.2:35361): tablet 34be0995612d4d4bb8103b81d56677e0 (table test-table [id=072c803cb25446afa9cff07df33431ea]) successfully deleted
W20260929 19:03:54.155632 16299 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:42445 (error 108)
W20260929 19:03:54.156061 16778 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:03:54.155940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42445 (local address 127.0.0.1:37422)
0929 19:03:54.155998 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0929 19:03:54.156031 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42445: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":39}
I20260929 19:03:54.156252 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUbnLHE/build/release/bin/kudu with pid 16506
W20260929 19:03:54.156158 16299 catalog_manager.cc:4584] TS 45494feaa6df4f32807cb11e2b10a536 (127.0.98.1:42445): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 34be0995612d4d4bb8103b81d56677e0: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42445: connect: Connection reset by peer (error 104)
I20260929 19:03:54.159879 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUbnLHE/build/release/bin/kudu with pid 16643
I20260929 19:03:54.163864 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUbnLHE/build/release/bin/kudu with pid 16281
I20260929 19:04:14.168721 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUbnLHE/build/release/bin/kudu
/tmp/dist-test-taskUbnLHE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40647
--webserver_interface=127.0.98.62
--webserver_port=34769
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40647
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:04:14.245966 16803 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:04:14.246141 16803 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:04:14.246163 16803 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:04:14.247231 16803 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260929 19:04:14.247277 16803 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260929 19:04:14.247289 16803 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260929 19:04:14.247324 16803 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 19:04:14.247355 16803 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:04:14.247373 16803 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:04:14.247399 16803 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 19:04:14.247429 16803 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 19:04:14.248808 16803 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40647
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40647
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=34769
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16803
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 18:09:13 UTC on ccycloud.quasar-hpkalf.root.comops.site
build id 4680913
I20260929 19:04:14.249048 16803 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:04:14.249248 16803 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 19:04:14.254356 16803 server_base.cc:1034] running on GCE node
W20260929 19:04:14.254567 16809 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:04:14.254520 16811 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:04:14.254523 16808 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:04:14.254992 16803 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:04:14.255280 16803 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:04:14.256431 16803 hybrid_clock.cc:648] HybridClock initialized: now 1790708654256424 us; error 38 us; skew 500 ppm
Sep 29 19:04:14 dist-test-slave-lqkv krb5kdc[15736](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708654, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:04:14.258796 16803 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 19:04:14.259176 16803 webserver.cc:460] Webserver started at http://127.0.98.62:34769/ using document root <none> and password file <none>
I20260929 19:04:14.259387 16803 fs_manager.cc:359] Metadata directory not provided
I20260929 19:04:14.259449 16803 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:04:14.260668 16803 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260929 19:04:14.261363 16818 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:04:14.261523 16803 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260929 19:04:14.261606 16803 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/minicluster-data/master-0/wal
uuid: "37a91a7695be4407b44d568eae618e7b"
format_stamp: "Formatted at 2026-09-29 19:03:32 on dist-test-slave-lqkv"
server_key: "65e3240cbb4140095e76fd16e533939a"
server_key_iv: "6e817c256fb43c0a2df9f04fa59f5a39"
server_key_version: "encryptionkey@0"
I20260929 19:04:14.261925 16803 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:04:14.277956 16803 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:04:14.279600 16803 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:04:14.279744 16803 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:04:14.283236 16803 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40647
I20260929 19:04:14.283303 16870 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40647 every 8 connection(s)
I20260929 19:04:14.283613 16803 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/minicluster-data/master-0/data/info.pb
I20260929 19:04:14.284999 16871 sys_catalog.cc:263] Verifying existing consensus state
I20260929 19:04:14.285601 16871 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37a91a7695be4407b44d568eae618e7b: Bootstrap starting.
I20260929 19:04:14.293396 16871 log.cc:824] T 00000000000000000000000000000000 P 37a91a7695be4407b44d568eae618e7b: Log is configured to *not* fsync() on all Append() calls
I20260929 19:04:14.293452 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUbnLHE/build/release/bin/kudu as pid 16803
I20260929 19:04:14.293740 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUbnLHE/build/release/bin/kudu
/tmp/dist-test-taskUbnLHE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:42445
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:40647
--webserver_port=41849
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 19:04:14.295783 16871 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37a91a7695be4407b44d568eae618e7b: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 19:04:14.296041 16871 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37a91a7695be4407b44d568eae618e7b: Bootstrap complete.
I20260929 19:04:14.298029 16871 raft_consensus.cc:348] T 00000000000000000000000000000000 P 37a91a7695be4407b44d568eae618e7b [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37a91a7695be4407b44d568eae618e7b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40647 } }
I20260929 19:04:14.298213 16871 raft_consensus.cc:729] T 00000000000000000000000000000000 P 37a91a7695be4407b44d568eae618e7b [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37a91a7695be4407b44d568eae618e7b, State: Initialized, Role: FOLLOWER
I20260929 19:04:14.298362 16871 consensus_queue.cc:260] T 00000000000000000000000000000000 P 37a91a7695be4407b44d568eae618e7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37a91a7695be4407b44d568eae618e7b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40647 } }
I20260929 19:04:14.298413 16871 raft_consensus.cc:388] T 00000000000000000000000000000000 P 37a91a7695be4407b44d568eae618e7b [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 19:04:14.298465 16871 raft_consensus.cc:482] T 00000000000000000000000000000000 P 37a91a7695be4407b44d568eae618e7b [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 19:04:14.298506 16871 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 37a91a7695be4407b44d568eae618e7b [term 2 FOLLOWER]: Advancing to term 3
I20260929 19:04:14.299408 16871 raft_consensus.cc:504] T 00000000000000000000000000000000 P 37a91a7695be4407b44d568eae618e7b [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37a91a7695be4407b44d568eae618e7b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40647 } }
I20260929 19:04:14.299486 16871 leader_election.cc:302] T 00000000000000000000000000000000 P 37a91a7695be4407b44d568eae618e7b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 37a91a7695be4407b44d568eae618e7b; no voters:
I20260929 19:04:14.299649 16871 leader_election.cc:288] T 00000000000000000000000000000000 P 37a91a7695be4407b44d568eae618e7b [CANDIDATE]: Term 3 election: Requested vote from peers
I20260929 19:04:14.299681 16875 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 37a91a7695be4407b44d568eae618e7b [term 3 FOLLOWER]: Leader election won for term 3
I20260929 19:04:14.299791 16875 raft_consensus.cc:686] T 00000000000000000000000000000000 P 37a91a7695be4407b44d568eae618e7b [term 3 LEADER]: Becoming Leader. State: Replica: 37a91a7695be4407b44d568eae618e7b, State: Running, Role: LEADER
I20260929 19:04:14.299896 16871 sys_catalog.cc:564] T 00000000000000000000000000000000 P 37a91a7695be4407b44d568eae618e7b [sys.catalog]: configured and running, proceeding with master startup.
I20260929 19:04:14.299930 16875 consensus_queue.cc:237] T 00000000000000000000000000000000 P 37a91a7695be4407b44d568eae618e7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37a91a7695be4407b44d568eae618e7b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40647 } }
I20260929 19:04:14.300177 16875 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37a91a7695be4407b44d568eae618e7b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 37a91a7695be4407b44d568eae618e7b. Latest consensus state: current_term: 3 leader_uuid: "37a91a7695be4407b44d568eae618e7b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37a91a7695be4407b44d568eae618e7b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40647 } } }
I20260929 19:04:14.300253 16875 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37a91a7695be4407b44d568eae618e7b [sys.catalog]: This master's current role is: LEADER
I20260929 19:04:14.300181 16877 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37a91a7695be4407b44d568eae618e7b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "37a91a7695be4407b44d568eae618e7b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37a91a7695be4407b44d568eae618e7b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40647 } } }
I20260929 19:04:14.300428 16877 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37a91a7695be4407b44d568eae618e7b [sys.catalog]: This master's current role is: LEADER
I20260929 19:04:14.300441 16880 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 19:04:14.301437 16880 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 19:04:14.301541 16880 catalog_manager.cc:1259] Loaded cluster ID: 487198941692428cb04133850e10e3ad
I20260929 19:04:14.301606 16880 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 19:04:14.302492 16880 catalog_manager.cc:1504] Loading token signing keys...
I20260929 19:04:14.302690 16880 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 37a91a7695be4407b44d568eae618e7b: Loaded TSK: 0
I20260929 19:04:14.302745 16880 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 37a91a7695be4407b44d568eae618e7b: Loaded TSK: 1
F20260929 19:04:14.302942 16880 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f9d15bf39ad google::LogMessage::Fail() at ??:0
@ 0x7f9d15bf688a google::LogMessage::SendToLog() at ??:0
@ 0x7f9d15bf34de google::LogMessage::Flush() at ??:0
@ 0x7f9d15bf4a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f9d18360104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f9d1835dd41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f9d1db4d1cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f9d1db4d48e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f9d1db4d594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f9d1db13c48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f9d1db1dad0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f9d17ac10d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f9d17ab6954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f9d1891a6db start_thread at ??:0
@ 0x7f9d146f161f clone at ??:0
W20260929 19:04:14.389595 16874 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:04:14.389771 16874 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:04:14.389792 16874 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:04:14.390885 16874 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:04:14.390923 16874 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:04:14.390959 16874 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 19:04:14.392328 16874 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:42445
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=41849
--tserver_master_addrs=127.0.98.62:40647
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16874
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 18:09:13 UTC on ccycloud.quasar-hpkalf.root.comops.site
build id 4680913
I20260929 19:04:14.392535 16874 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:04:14.392747 16874 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:04:14.398689 16913 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:04:14.398689 16915 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:04:14.398881 16874 server_base.cc:1034] running on GCE node
W20260929 19:04:14.398907 16912 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:04:14.399148 16874 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:04:14.399379 16874 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:04:14.400530 16874 hybrid_clock.cc:648] HybridClock initialized: now 1790708654400519 us; error 29 us; skew 500 ppm
Sep 29 19:04:14 dist-test-slave-lqkv krb5kdc[15736](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708654, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:04:14.403043 16874 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 19:04:14.403503 16874 webserver.cc:460] Webserver started at http://127.0.98.1:41849/ using document root <none> and password file <none>
I20260929 19:04:14.403714 16874 fs_manager.cc:359] Metadata directory not provided
I20260929 19:04:14.403779 16874 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:04:14.405129 16874 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260929 19:04:14.405821 16922 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:04:14.405973 16874 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 19:04:14.406057 16874 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/minicluster-data/ts-0/wal
uuid: "45494feaa6df4f32807cb11e2b10a536"
format_stamp: "Formatted at 2026-09-29 19:03:32 on dist-test-slave-lqkv"
server_key: "a7f246d56390ed72651d42d8a5f3c919"
server_key_iv: "0d2f63a0d7078e0d0ae5e7d52d9e9635"
server_key_version: "encryptionkey@0"
I20260929 19:04:14.406364 16874 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:04:14.413013 16874 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:04:14.414911 16874 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:04:14.415058 16874 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:04:14.415264 16874 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:04:14.415638 16929 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260929 19:04:14.417320 16874 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260929 19:04:14.417374 16874 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260929 19:04:14.417414 16874 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260929 19:04:14.417966 16874 ts_tablet_manager.cc:610] Registered 1 tablets
I20260929 19:04:14.418016 16874 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260929 19:04:14.418079 16929 tablet_bootstrap.cc:492] T 34be0995612d4d4bb8103b81d56677e0 P 45494feaa6df4f32807cb11e2b10a536: Bootstrap starting.
I20260929 19:04:14.424995 16874 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42445
I20260929 19:04:14.425117 17036 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42445 every 8 connection(s)
I20260929 19:04:14.425412 16874 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/minicluster-data/ts-0/data/info.pb
I20260929 19:04:14.428512 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUbnLHE/build/release/bin/kudu as pid 16874
I20260929 19:04:14.428706 16929 log.cc:824] T 34be0995612d4d4bb8103b81d56677e0 P 45494feaa6df4f32807cb11e2b10a536: Log is configured to *not* fsync() on all Append() calls
I20260929 19:04:14.428874 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUbnLHE/build/release/bin/kudu
/tmp/dist-test-taskUbnLHE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:35361
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:40647
--webserver_port=37303
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 19:04:14.431463 16929 tablet_bootstrap.cc:492] T 34be0995612d4d4bb8103b81d56677e0 P 45494feaa6df4f32807cb11e2b10a536: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 19:04:14.431749 16929 tablet_bootstrap.cc:492] T 34be0995612d4d4bb8103b81d56677e0 P 45494feaa6df4f32807cb11e2b10a536: Bootstrap complete.
I20260929 19:04:14.432904 16929 ts_tablet_manager.cc:1397] T 34be0995612d4d4bb8103b81d56677e0 P 45494feaa6df4f32807cb11e2b10a536: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.010s
I20260929 19:04:14.434274 16929 raft_consensus.cc:348] T 34be0995612d4d4bb8103b81d56677e0 P 45494feaa6df4f32807cb11e2b10a536 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43c7b519591f4119a105016fef7c4656" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44775 } } peers { permanent_uuid: "c8150c87f2684d4c937a086a3ef65e2a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35361 } } peers { permanent_uuid: "45494feaa6df4f32807cb11e2b10a536" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42445 } }
I20260929 19:04:14.434590 16929 raft_consensus.cc:729] T 34be0995612d4d4bb8103b81d56677e0 P 45494feaa6df4f32807cb11e2b10a536 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45494feaa6df4f32807cb11e2b10a536, State: Initialized, Role: FOLLOWER
I20260929 19:04:14.434749 16929 consensus_queue.cc:260] T 34be0995612d4d4bb8103b81d56677e0 P 45494feaa6df4f32807cb11e2b10a536 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43c7b519591f4119a105016fef7c4656" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44775 } } peers { permanent_uuid: "c8150c87f2684d4c937a086a3ef65e2a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35361 } } peers { permanent_uuid: "45494feaa6df4f32807cb11e2b10a536" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42445 } }
I20260929 19:04:14.435034 16929 ts_tablet_manager.cc:1428] T 34be0995612d4d4bb8103b81d56677e0 P 45494feaa6df4f32807cb11e2b10a536: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
W20260929 19:04:14.504139 17039 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:04:14.425967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40647 (local address 127.0.98.1:48913)
0929 19:04:14.426108 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0929 19:04:14.426144 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0929 19:04:14.426553 (+ 409us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:04:14.503917 (+ 77364us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40647: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":84,"threads_started":1}
W20260929 19:04:14.504448 17037 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:40647 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:40647: Client connection negotiation failed: client connection to 127.0.98.62:40647: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 19:04:14.504858 17039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:14.504696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40647 (local address 127.0.98.1:40621)
0929 19:04:14.504756 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0929 19:04:14.504832 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260929 19:04:14.505190 17039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:14.505090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40647 (local address 127.0.98.1:41097)
0929 19:04:14.505139 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0929 19:04:14.505169 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260929 19:04:14.505335 17037 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 19:04:14.517835 17041 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:04:14.518006 17041 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:04:14.518028 17041 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:04:14.519171 17041 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:04:14.519214 17041 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:04:14.519265 17041 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 19:04:14.520597 17041 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:35361
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=37303
--tserver_master_addrs=127.0.98.62:40647
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17041
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 18:09:13 UTC on ccycloud.quasar-hpkalf.root.comops.site
build id 4680913
I20260929 19:04:14.520845 17041 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:04:14.521060 17041 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:04:14.525887 17050 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:04:14.526026 17041 server_base.cc:1034] running on GCE node
W20260929 19:04:14.525873 17048 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:04:14.525928 17047 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:04:14.526275 17041 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:04:14.526501 17041 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:04:14.527660 17041 hybrid_clock.cc:648] HybridClock initialized: now 1790708654527647 us; error 23 us; skew 500 ppm
Sep 29 19:04:14 dist-test-slave-lqkv krb5kdc[15736](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708654, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:04:14.529806 17041 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 19:04:14.530217 17041 webserver.cc:460] Webserver started at http://127.0.98.2:37303/ using document root <none> and password file <none>
I20260929 19:04:14.530413 17041 fs_manager.cc:359] Metadata directory not provided
I20260929 19:04:14.530478 17041 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:04:14.531603 17041 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 19:04:14.532128 17057 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:04:14.532271 17041 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 19:04:14.532356 17041 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/minicluster-data/ts-1/wal
uuid: "c8150c87f2684d4c937a086a3ef65e2a"
format_stamp: "Formatted at 2026-09-29 19:03:33 on dist-test-slave-lqkv"
server_key: "e7b0ad82b16a7d630e34d8eebaf9958f"
server_key_iv: "076a095a7e7f74a0625173fc5676be78"
server_key_version: "encryptionkey@0"
I20260929 19:04:14.532650 17041 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:04:14.563241 17041 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:04:14.565204 17041 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:04:14.565388 17041 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:04:14.565632 17041 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:04:14.565927 17041 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:04:14.565979 17041 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:04:14.566013 17041 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:04:14.566039 17041 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:04:14.571571 17041 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35361
I20260929 19:04:14.571631 17170 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35361 every 8 connection(s)
I20260929 19:04:14.571943 17041 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/minicluster-data/ts-1/data/info.pb
I20260929 19:04:14.572973 17173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:14.572547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40647 (local address 127.0.98.2:59763)
0929 19:04:14.572706 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0929 19:04:14.572769 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":89,"threads_started":1}
W20260929 19:04:14.573362 17171 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:40647 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:40647: Client connection negotiation failed: client connection to 127.0.98.62:40647: connect: Connection refused (error 111)
I20260929 19:04:14.573638 17173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:14.573546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40647 (local address 127.0.98.2:36447)
0929 19:04:14.573585 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0929 19:04:14.573622 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260929 19:04:14.573817 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUbnLHE/build/release/bin/kudu as pid 17041
I20260929 19:04:14.573982 17173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:14.573871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40647 (local address 127.0.98.2:42205)
0929 19:04:14.573917 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0929 19:04:14.573966 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260929 19:04:14.574134 17171 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 19:04:14.574199 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUbnLHE/build/release/bin/kudu
/tmp/dist-test-taskUbnLHE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:44775
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:40647
--webserver_port=43749
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:04:14.650851 17174 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:04:14.651018 17174 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:04:14.651058 17174 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:04:14.652427 17174 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:04:14.652473 17174 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:04:14.652501 17174 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 19:04:14.654027 17174 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:44775
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=43749
--tserver_master_addrs=127.0.98.62:40647
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17174
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 18:09:13 UTC on ccycloud.quasar-hpkalf.root.comops.site
build id 4680913
I20260929 19:04:14.654222 17174 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:04:14.654387 17174 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:04:14.659142 17180 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:04:14.659159 17182 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:04:14.659299 17174 server_base.cc:1034] running on GCE node
W20260929 19:04:14.659152 17179 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:04:14.659593 17174 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:04:14.659832 17174 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:04:14.660993 17174 hybrid_clock.cc:648] HybridClock initialized: now 1790708654660982 us; error 23 us; skew 500 ppm
Sep 29 19:04:14 dist-test-slave-lqkv krb5kdc[15736](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708654, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:04:14.663280 17174 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 19:04:14.663672 17174 webserver.cc:460] Webserver started at http://127.0.98.3:43749/ using document root <none> and password file <none>
I20260929 19:04:14.663898 17174 fs_manager.cc:359] Metadata directory not provided
I20260929 19:04:14.663965 17174 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:04:14.665256 17174 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 19:04:14.665874 17189 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:04:14.666035 17174 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 19:04:14.666127 17174 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/minicluster-data/ts-2/wal
uuid: "43c7b519591f4119a105016fef7c4656"
format_stamp: "Formatted at 2026-09-29 19:03:33 on dist-test-slave-lqkv"
server_key: "9dbbaaf51b9418e023cc82e84ca5e550"
server_key_iv: "c339eb06ae9f36bf9a77ec5eb2974423"
server_key_version: "encryptionkey@0"
I20260929 19:04:14.666430 17174 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:04:14.680637 17174 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:04:14.682360 17174 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:04:14.682511 17174 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:04:14.682708 17174 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:04:14.682981 17174 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:04:14.683033 17174 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:04:14.683072 17174 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:04:14.683099 17174 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:04:14.689086 17174 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44775
I20260929 19:04:14.689118 17302 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44775 every 8 connection(s)
I20260929 19:04:14.689440 17174 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790708447806542-392-0/minicluster-data/ts-2/data/info.pb
I20260929 19:04:14.690385 17305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:14.690065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40647 (local address 127.0.98.3:38171)
0929 19:04:14.690195 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0929 19:04:14.690254 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":64,"threads_started":1}
W20260929 19:04:14.690761 17303 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:40647 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:40647: Client connection negotiation failed: client connection to 127.0.98.62:40647: connect: Connection refused (error 111)
I20260929 19:04:14.691004 17305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:14.690922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40647 (local address 127.0.98.3:58513)
0929 19:04:14.690963 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0929 19:04:14.690987 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260929 19:04:14.691260 17305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:14.691193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40647 (local address 127.0.98.3:39997)
0929 19:04:14.691228 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0929 19:04:14.691248 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260929 19:04:14.691375 17303 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 19:04:14.698591 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUbnLHE/build/release/bin/kudu as pid 17174
I20260929 19:04:15.505853 17307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:15.505667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40647 (local address 127.0.98.1:60547)
0929 19:04:15.505763 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0929 19:04:15.505795 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81,"thread_start_us":53,"threads_started":1}
I20260929 19:04:15.574605 17308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:15.574416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40647 (local address 127.0.98.2:36793)
0929 19:04:15.574522 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0929 19:04:15.574553 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":50,"threads_started":1}
I20260929 19:04:15.691888 17309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:15.691675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40647 (local address 127.0.98.3:52475)
0929 19:04:15.691818 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0929 19:04:15.691845 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":100,"threads_started":1}
I20260929 19:04:16.032888 17310 raft_consensus.cc:482] T 34be0995612d4d4bb8103b81d56677e0 P 45494feaa6df4f32807cb11e2b10a536 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 19:04:16.032994 17310 raft_consensus.cc:504] T 34be0995612d4d4bb8103b81d56677e0 P 45494feaa6df4f32807cb11e2b10a536 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43c7b519591f4119a105016fef7c4656" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44775 } } peers { permanent_uuid: "c8150c87f2684d4c937a086a3ef65e2a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35361 } } peers { permanent_uuid: "45494feaa6df4f32807cb11e2b10a536" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42445 } }
I20260929 19:04:16.033232 17310 leader_election.cc:288] T 34be0995612d4d4bb8103b81d56677e0 P 45494feaa6df4f32807cb11e2b10a536 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 43c7b519591f4119a105016fef7c4656 (127.0.98.3:44775), c8150c87f2684d4c937a086a3ef65e2a (127.0.98.2:35361)
Sep 29 19:04:16 dist-test-slave-lqkv krb5kdc[15736](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708654, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
Sep 29 19:04:16 dist-test-slave-lqkv krb5kdc[15736](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708654, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
I20260929 19:04:16.042879 17311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:16.033399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44775 (local address 127.0.98.1:34953)
0929 19:04:16.033500 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0929 19:04:16.033509 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 19:04:16.033559 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:04:16.034457 (+ 898us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:04:16.034460 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:04:16.034856 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:04:16.035069 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:04:16.035075 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:04:16.036172 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:04:16.036173 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:04:16.036956 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:04:16.036958 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:04:16.037501 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:04:16.037502 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:04:16.037590 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:04:16.038150 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:04:16.038155 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:04:16.039442 (+ 1287us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:04:16.041444 (+ 2002us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:04:16.041450 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:04:16.041452 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:04:16.041681 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:04:16.041948 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:04:16.041949 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:04:16.041949 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:04:16.042055 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:04:16.042409 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:04:16.042414 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:04:16.042529 (+ 115us) client_negotiation.cc:769] Sending connection context
0929 19:04:16.042616 (+ 87us) client_negotiation.cc:240] Negotiation successful
0929 19:04:16.042792 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":82,"mutex_wait_us":36,"thread_start_us":63,"threads_started":1}
I20260929 19:04:16.042888 17312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:16.033477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34953 (local address 127.0.98.3:44775)
0929 19:04:16.033791 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0929 19:04:16.033801 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 19:04:16.033830 (+ 29us) server_negotiation.cc:407] Connection header received
0929 19:04:16.034242 (+ 412us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:04:16.034244 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:04:16.034288 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:04:16.034415 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:04:16.035190 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:16.036096 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:16.037029 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:16.037403 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:16.037506 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:04:16.039548 (+ 2042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:04:16.039551 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:04:16.039552 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:04:16.039562 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:04:16.041350 (+ 1788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:04:16.041773 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:04:16.041775 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:04:16.041776 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:04:16.041864 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:04:16.042133 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:04:16.042134 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:04:16.042134 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:04:16.042342 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:04:16.042425 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0929 19:04:16.042666 (+ 241us) server_negotiation.cc:299] Negotiation successful
0929 19:04:16.042783 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":71,"threads_started":1}
I20260929 19:04:16.042923 17314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:16.033648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37391 (local address 127.0.98.2:35361)
0929 19:04:16.033793 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0929 19:04:16.033808 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 19:04:16.033833 (+ 25us) server_negotiation.cc:407] Connection header received
0929 19:04:16.034242 (+ 409us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:04:16.034244 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:04:16.034283 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:04:16.034422 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:04:16.035180 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:16.036063 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:16.037038 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:16.037400 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:16.037505 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:04:16.039909 (+ 2404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:04:16.039912 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:04:16.039913 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:04:16.039921 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:04:16.041415 (+ 1494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:04:16.041786 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:04:16.041789 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:04:16.041789 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:04:16.041912 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:04:16.042133 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:04:16.042134 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:04:16.042134 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:04:16.042344 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:04:16.042425 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 19:04:16.042675 (+ 250us) server_negotiation.cc:299] Negotiation successful
0929 19:04:16.042820 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":50,"threads_started":1}
I20260929 19:04:16.042987 17313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:16.033558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35361 (local address 127.0.98.1:37391)
0929 19:04:16.033648 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0929 19:04:16.033655 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0929 19:04:16.033693 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:04:16.034512 (+ 819us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:04:16.034513 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:04:16.034869 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:04:16.035071 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:04:16.035076 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:04:16.036150 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:04:16.036151 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:04:16.036958 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:04:16.036960 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:04:16.037490 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:04:16.037491 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:04:16.037595 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:04:16.038102 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:04:16.038109 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:04:16.039800 (+ 1691us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:04:16.041492 (+ 1692us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:04:16.041493 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:04:16.041494 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:04:16.041704 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:04:16.041983 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:04:16.041984 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:04:16.041985 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:04:16.042071 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:04:16.042419 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:04:16.042420 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:04:16.042530 (+ 110us) client_negotiation.cc:769] Sending connection context
0929 19:04:16.042616 (+ 86us) client_negotiation.cc:240] Negotiation successful
0929 19:04:16.042865 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":81,"mutex_wait_us":213,"thread_start_us":38,"threads_started":1}
W20260929 19:04:16.044147 16925 leader_election.cc:341] T 34be0995612d4d4bb8103b81d56677e0 P 45494feaa6df4f32807cb11e2b10a536 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c8150c87f2684d4c937a086a3ef65e2a (127.0.98.2:35361): Not found: Tablet not found: 34be0995612d4d4bb8103b81d56677e0
W20260929 19:04:16.044215 16925 leader_election.cc:341] T 34be0995612d4d4bb8103b81d56677e0 P 45494feaa6df4f32807cb11e2b10a536 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 43c7b519591f4119a105016fef7c4656 (127.0.98.3:44775): Not found: Tablet not found: 34be0995612d4d4bb8103b81d56677e0
I20260929 19:04:16.044255 16925 leader_election.cc:302] T 34be0995612d4d4bb8103b81d56677e0 P 45494feaa6df4f32807cb11e2b10a536 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 45494feaa6df4f32807cb11e2b10a536; no voters: 43c7b519591f4119a105016fef7c4656, c8150c87f2684d4c937a086a3ef65e2a
I20260929 19:04:16.044350 17310 raft_consensus.cc:2726] T 34be0995612d4d4bb8103b81d56677e0 P 45494feaa6df4f32807cb11e2b10a536 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 19:04:16.506366 17313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:16.506235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40647 (local address 127.0.98.1:37255)
0929 19:04:16.506284 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0929 19:04:16.506326 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
I20260929 19:04:16.575155 17315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:16.574987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40647 (local address 127.0.98.2:60199)
0929 19:04:16.575090 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0929 19:04:16.575121 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":52,"threads_started":1}
I20260929 19:04:16.692422 17316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:16.692235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40647 (local address 127.0.98.3:54897)
0929 19:04:16.692359 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0929 19:04:16.692388 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":72,"threads_started":1}
I20260929 19:04:17.507035 17317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:17.506743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40647 (local address 127.0.98.1:60779)
0929 19:04:17.506956 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0929 19:04:17.506990 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":53,"threads_started":1}
I20260929 19:04:17.575726 17318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:17.575538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40647 (local address 127.0.98.2:35895)
0929 19:04:17.575667 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0929 19:04:17.575697 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":69,"threads_started":1}
I20260929 19:04:17.692992 17319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:17.692786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40647 (local address 127.0.98.3:35921)
0929 19:04:17.692930 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0929 19:04:17.692959 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":97,"threads_started":1}
I20260929 19:04:17.990006 17320 raft_consensus.cc:482] T 34be0995612d4d4bb8103b81d56677e0 P 45494feaa6df4f32807cb11e2b10a536 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 19:04:17.990079 17320 raft_consensus.cc:504] T 34be0995612d4d4bb8103b81d56677e0 P 45494feaa6df4f32807cb11e2b10a536 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43c7b519591f4119a105016fef7c4656" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44775 } } peers { permanent_uuid: "c8150c87f2684d4c937a086a3ef65e2a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35361 } } peers { permanent_uuid: "45494feaa6df4f32807cb11e2b10a536" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42445 } }
I20260929 19:04:17.990252 17320 leader_election.cc:288] T 34be0995612d4d4bb8103b81d56677e0 P 45494feaa6df4f32807cb11e2b10a536 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 43c7b519591f4119a105016fef7c4656 (127.0.98.3:44775), c8150c87f2684d4c937a086a3ef65e2a (127.0.98.2:35361)
W20260929 19:04:17.990618 16925 leader_election.cc:341] T 34be0995612d4d4bb8103b81d56677e0 P 45494feaa6df4f32807cb11e2b10a536 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 43c7b519591f4119a105016fef7c4656 (127.0.98.3:44775): Not found: Tablet not found: 34be0995612d4d4bb8103b81d56677e0
W20260929 19:04:17.990687 16925 leader_election.cc:341] T 34be0995612d4d4bb8103b81d56677e0 P 45494feaa6df4f32807cb11e2b10a536 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c8150c87f2684d4c937a086a3ef65e2a (127.0.98.2:35361): Not found: Tablet not found: 34be0995612d4d4bb8103b81d56677e0
I20260929 19:04:17.990702 16925 leader_election.cc:302] T 34be0995612d4d4bb8103b81d56677e0 P 45494feaa6df4f32807cb11e2b10a536 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 45494feaa6df4f32807cb11e2b10a536; no voters: 43c7b519591f4119a105016fef7c4656, c8150c87f2684d4c937a086a3ef65e2a
I20260929 19:04:17.990762 17320 raft_consensus.cc:2726] T 34be0995612d4d4bb8103b81d56677e0 P 45494feaa6df4f32807cb11e2b10a536 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 19:04:18.507820 17321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:18.507623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40647 (local address 127.0.98.1:51587)
0929 19:04:18.507740 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0929 19:04:18.507772 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":56,"threads_started":1}
I20260929 19:04:18.576423 17322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:18.576229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40647 (local address 127.0.98.2:55705)
0929 19:04:18.576351 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0929 19:04:18.576388 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":69,"threads_started":1}
I20260929 19:04:18.693583 17323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:18.693399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40647 (local address 127.0.98.3:46029)
0929 19:04:18.693518 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0929 19:04:18.693547 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":70,"threads_started":1}
I20260929 19:04:19.508487 17324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:19.508278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40647 (local address 127.0.98.1:59525)
0929 19:04:19.508409 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0929 19:04:19.508442 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":56,"threads_started":1}
I20260929 19:04:19.577008 17325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:19.576838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40647 (local address 127.0.98.2:51055)
0929 19:04:19.576943 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0929 19:04:19.576974 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":54,"threads_started":1}
I20260929 19:04:19.694152 17326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:19.693976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40647 (local address 127.0.98.3:54623)
0929 19:04:19.694087 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0929 19:04:19.694119 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":52,"threads_started":1}
I20260929 19:04:20.509065 17327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:20.508884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40647 (local address 127.0.98.1:49213)
0929 19:04:20.508990 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0929 19:04:20.509019 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":52,"threads_started":1}
I20260929 19:04:20.577611 17328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:20.577381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40647 (local address 127.0.98.2:46029)
0929 19:04:20.577530 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0929 19:04:20.577561 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":76,"threads_started":1}
I20260929 19:04:20.694730 17329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:20.694530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40647 (local address 127.0.98.3:55425)
0929 19:04:20.694654 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0929 19:04:20.694683 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":69,"threads_started":1}
I20260929 19:04:20.924989 17330 raft_consensus.cc:482] T 34be0995612d4d4bb8103b81d56677e0 P 45494feaa6df4f32807cb11e2b10a536 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 19:04:20.925091 17330 raft_consensus.cc:504] T 34be0995612d4d4bb8103b81d56677e0 P 45494feaa6df4f32807cb11e2b10a536 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43c7b519591f4119a105016fef7c4656" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44775 } } peers { permanent_uuid: "c8150c87f2684d4c937a086a3ef65e2a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35361 } } peers { permanent_uuid: "45494feaa6df4f32807cb11e2b10a536" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42445 } }
I20260929 19:04:20.925261 17330 leader_election.cc:288] T 34be0995612d4d4bb8103b81d56677e0 P 45494feaa6df4f32807cb11e2b10a536 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 43c7b519591f4119a105016fef7c4656 (127.0.98.3:44775), c8150c87f2684d4c937a086a3ef65e2a (127.0.98.2:35361)
W20260929 19:04:20.925622 16925 leader_election.cc:341] T 34be0995612d4d4bb8103b81d56677e0 P 45494feaa6df4f32807cb11e2b10a536 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 43c7b519591f4119a105016fef7c4656 (127.0.98.3:44775): Not found: Tablet not found: 34be0995612d4d4bb8103b81d56677e0
W20260929 19:04:20.925694 16925 leader_election.cc:341] T 34be0995612d4d4bb8103b81d56677e0 P 45494feaa6df4f32807cb11e2b10a536 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c8150c87f2684d4c937a086a3ef65e2a (127.0.98.2:35361): Not found: Tablet not found: 34be0995612d4d4bb8103b81d56677e0
I20260929 19:04:20.925741 16925 leader_election.cc:302] T 34be0995612d4d4bb8103b81d56677e0 P 45494feaa6df4f32807cb11e2b10a536 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 45494feaa6df4f32807cb11e2b10a536; no voters: 43c7b519591f4119a105016fef7c4656, c8150c87f2684d4c937a086a3ef65e2a
I20260929 19:04:20.925809 17330 raft_consensus.cc:2726] T 34be0995612d4d4bb8103b81d56677e0 P 45494feaa6df4f32807cb11e2b10a536 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 19:04:21.509608 17331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:21.509426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40647 (local address 127.0.98.1:39903)
0929 19:04:21.509533 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0929 19:04:21.509562 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":54,"threads_started":1}
I20260929 19:04:21.578166 17332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:21.577998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40647 (local address 127.0.98.2:46191)
0929 19:04:21.578103 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0929 19:04:21.578132 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":54,"threads_started":1}
I20260929 19:04:21.695303 17333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:21.695115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40647 (local address 127.0.98.3:45891)
0929 19:04:21.695237 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0929 19:04:21.695269 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":66,"threads_started":1}
I20260929 19:04:22.510202 17334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:22.510003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40647 (local address 127.0.98.1:50261)
0929 19:04:22.510126 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0929 19:04:22.510158 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":53,"threads_started":1}
I20260929 19:04:22.578946 17335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:22.578751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40647 (local address 127.0.98.2:60629)
0929 19:04:22.578884 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0929 19:04:22.578916 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":69,"threads_started":1}
I20260929 19:04:22.695891 17336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:22.695700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40647 (local address 127.0.98.3:38779)
0929 19:04:22.695827 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0929 19:04:22.695858 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":54,"threads_started":1}
I20260929 19:04:23.510747 17337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:23.510562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40647 (local address 127.0.98.1:47735)
0929 19:04:23.510674 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0929 19:04:23.510699 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":58,"threads_started":1}
I20260929 19:04:23.579662 17338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:23.579460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40647 (local address 127.0.98.2:44425)
0929 19:04:23.579593 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0929 19:04:23.579628 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":54,"threads_started":1}
I20260929 19:04:23.696485 17339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:23.696269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40647 (local address 127.0.98.3:44547)
0929 19:04:23.696414 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0929 19:04:23.696451 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":69,"threads_started":1}
I20260929 19:04:24.511351 17340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:24.511126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40647 (local address 127.0.98.1:44371)
0929 19:04:24.511256 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0929 19:04:24.511293 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":64,"threads_started":1}
I20260929 19:04:24.580320 17341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:24.580105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40647 (local address 127.0.98.2:51879)
0929 19:04:24.580253 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0929 19:04:24.580288 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":111,"threads_started":1}
I20260929 19:04:24.697072 17342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:24.696907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40647 (local address 127.0.98.3:43231)
0929 19:04:24.697015 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0929 19:04:24.697041 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":68,"threads_started":1}
I20260929 19:04:25.511888 17343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:25.511687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40647 (local address 127.0.98.1:33349)
0929 19:04:25.511803 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0929 19:04:25.511842 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":58,"threads_started":1}
I20260929 19:04:25.581005 17344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:25.580829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40647 (local address 127.0.98.2:44557)
0929 19:04:25.580943 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0929 19:04:25.580974 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":63,"threads_started":1}
I20260929 19:04:25.587387 17345 raft_consensus.cc:482] T 34be0995612d4d4bb8103b81d56677e0 P 45494feaa6df4f32807cb11e2b10a536 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 19:04:25.587486 17345 raft_consensus.cc:504] T 34be0995612d4d4bb8103b81d56677e0 P 45494feaa6df4f32807cb11e2b10a536 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "43c7b519591f4119a105016fef7c4656" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44775 } } peers { permanent_uuid: "c8150c87f2684d4c937a086a3ef65e2a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35361 } } peers { permanent_uuid: "45494feaa6df4f32807cb11e2b10a536" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42445 } }
I20260929 19:04:25.587682 17345 leader_election.cc:288] T 34be0995612d4d4bb8103b81d56677e0 P 45494feaa6df4f32807cb11e2b10a536 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 43c7b519591f4119a105016fef7c4656 (127.0.98.3:44775), c8150c87f2684d4c937a086a3ef65e2a (127.0.98.2:35361)
W20260929 19:04:25.588047 16925 leader_election.cc:341] T 34be0995612d4d4bb8103b81d56677e0 P 45494feaa6df4f32807cb11e2b10a536 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 43c7b519591f4119a105016fef7c4656 (127.0.98.3:44775): Not found: Tablet not found: 34be0995612d4d4bb8103b81d56677e0
W20260929 19:04:25.588112 16925 leader_election.cc:341] T 34be0995612d4d4bb8103b81d56677e0 P 45494feaa6df4f32807cb11e2b10a536 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c8150c87f2684d4c937a086a3ef65e2a (127.0.98.2:35361): Not found: Tablet not found: 34be0995612d4d4bb8103b81d56677e0
I20260929 19:04:25.588127 16925 leader_election.cc:302] T 34be0995612d4d4bb8103b81d56677e0 P 45494feaa6df4f32807cb11e2b10a536 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 45494feaa6df4f32807cb11e2b10a536; no voters: 43c7b519591f4119a105016fef7c4656, c8150c87f2684d4c937a086a3ef65e2a
I20260929 19:04:25.588195 17345 raft_consensus.cc:2726] T 34be0995612d4d4bb8103b81d56677e0 P 45494feaa6df4f32807cb11e2b10a536 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 19:04:25.697625 17346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:25.697430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40647 (local address 127.0.98.3:38249)
0929 19:04:25.697546 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0929 19:04:25.697579 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":64,"threads_started":1}
I20260929 19:04:26.512459 17347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:26.512267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40647 (local address 127.0.98.1:45323)
0929 19:04:26.512382 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0929 19:04:26.512414 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":52,"threads_started":1}
I20260929 19:04:26.581645 17348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:26.581470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40647 (local address 127.0.98.2:52495)
0929 19:04:26.581580 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0929 19:04:26.581610 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":64,"threads_started":1}
I20260929 19:04:26.698201 17349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:26.697995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40647 (local address 127.0.98.3:41689)
0929 19:04:26.698112 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0929 19:04:26.698163 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":67,"threads_started":1}
I20260929 19:04:27.513074 17350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:27.512885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40647 (local address 127.0.98.1:40713)
0929 19:04:27.513003 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0929 19:04:27.513030 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":62,"threads_started":1}
I20260929 19:04:27.582254 17351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:27.581988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40647 (local address 127.0.98.2:50239)
0929 19:04:27.582173 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0929 19:04:27.582205 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":50,"threads_started":1}
I20260929 19:04:27.698944 17352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:27.698650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40647 (local address 127.0.98.3:38181)
0929 19:04:27.698879 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0929 19:04:27.698912 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":67,"threads_started":1}
I20260929 19:04:28.513659 17353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:28.513461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40647 (local address 127.0.98.1:54419)
0929 19:04:28.513586 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0929 19:04:28.513615 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":53,"threads_started":1}
I20260929 19:04:28.583031 17354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:28.582846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40647 (local address 127.0.98.2:39645)
0929 19:04:28.582968 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0929 19:04:28.582999 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":69,"threads_started":1}
I20260929 19:04:28.699496 17355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:28.699316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40647 (local address 127.0.98.3:58177)
0929 19:04:28.699427 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0929 19:04:28.699457 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":64,"threads_started":1}
I20260929 19:04:29.514285 17356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:29.514082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40647 (local address 127.0.98.1:51009)
0929 19:04:29.514211 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0929 19:04:29.514240 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":68,"threads_started":1}
I20260929 19:04:29.583794 17357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:29.583598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40647 (local address 127.0.98.2:53919)
0929 19:04:29.583735 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0929 19:04:29.583763 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":86,"threads_started":1}
I20260929 19:04:29.700179 17358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:29.699983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40647 (local address 127.0.98.3:57635)
0929 19:04:29.700109 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0929 19:04:29.700145 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40647: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":68,"threads_started":1}
I20260929 19:04:29.709174 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUbnLHE/build/release/bin/kudu with pid 16874
I20260929 19:04:29.713444 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUbnLHE/build/release/bin/kudu with pid 17041
I20260929 19:04:29.718273 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUbnLHE/build/release/bin/kudu with pid 17174
2026-09-29T19:04:29Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58950 ms)
[----------] 30 tests from SecurityITest (222735 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708447806542-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 19:04:30 dist-test-slave-lqkv krb5kdc[17361](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 19:04:30 dist-test-slave-lqkv krb5kdc[17361](info): set up 2 sockets
Sep 29 19:04:30 dist-test-slave-lqkv krb5kdc[17361](info): commencing operation
krb5kdc: starting...
W20260929 19:04:31.520257 392 mini_kdc.cc:121] Time spent starting KDC: real 0.965s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708447806542-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708447806542-392-0/krb5kdc/test-user.keytab.
Sep 29 19:04:31 dist-test-slave-lqkv krb5kdc[17361](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708671, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T19:04:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T19:04:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708447806542-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708447806542-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 19:04:31.623689 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUbnLHE/build/release/bin/kudu
/tmp/dist-test-taskUbnLHE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708447806542-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708447806542-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708447806542-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708447806542-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34385
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708447806542-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34385
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708447806542-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708447806542-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708447806542-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:04:31.703817 17374 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:04:31.704134 17374 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:04:31.704180 17374 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:04:31.705322 17374 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 19:04:31.705381 17374 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:04:31.705403 17374 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:04:31.705427 17374 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 19:04:31.705456 17374 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 19:04:31.706868 17374 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708447806542-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708447806542-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34385
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708447806542-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34385
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708447806542-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17374
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708447806542-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 18:09:13 UTC on ccycloud.quasar-hpkalf.root.comops.site
build id 4680913
I20260929 19:04:31.707120 17374 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:04:31.707370 17374 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:04:31.711912 17380 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:04:31.712065 17382 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:04:31.711947 17379 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:04:31.712183 17374 server_base.cc:1034] running on GCE node
I20260929 19:04:31.712535 17374 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:04:31.712865 17374 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:04:31.714015 17374 hybrid_clock.cc:648] HybridClock initialized: now 1790708671713997 us; error 35 us; skew 500 ppm
Sep 29 19:04:31 dist-test-slave-lqkv krb5kdc[17361](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708671, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:04:31.716274 17374 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 19:04:31.716620 17374 webserver.cc:460] Webserver started at http://127.0.98.62:43149/ using document root <none> and password file <none>
I20260929 19:04:31.716796 17374 fs_manager.cc:359] Metadata directory not provided
I20260929 19:04:31.716905 17374 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:04:31.716991 17374 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:04:31.717940 17374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708447806542-392-0/minicluster-data/master-0/data/instance:
uuid: "5eae6349988c4802981353bc907e10d5"
format_stamp: "Formatted at 2026-09-29 19:04:31 on dist-test-slave-lqkv"
server_key: "74b87ee87bf8b5b11c7941d711743518"
server_key_iv: "9c4d1f5ecbbd2472ab5d3ae4343e97f4"
server_key_version: "encryptionkey@0"
I20260929 19:04:31.718220 17374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708447806542-392-0/minicluster-data/master-0/wal/instance:
uuid: "5eae6349988c4802981353bc907e10d5"
format_stamp: "Formatted at 2026-09-29 19:04:31 on dist-test-slave-lqkv"
server_key: "74b87ee87bf8b5b11c7941d711743518"
server_key_iv: "9c4d1f5ecbbd2472ab5d3ae4343e97f4"
server_key_version: "encryptionkey@0"
I20260929 19:04:31.719378 17374 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260929 19:04:31.719990 17389 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:04:31.720134 17374 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 19:04:31.720208 17374 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708447806542-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708447806542-392-0/minicluster-data/master-0/wal
uuid: "5eae6349988c4802981353bc907e10d5"
format_stamp: "Formatted at 2026-09-29 19:04:31 on dist-test-slave-lqkv"
server_key: "74b87ee87bf8b5b11c7941d711743518"
server_key_iv: "9c4d1f5ecbbd2472ab5d3ae4343e97f4"
server_key_version: "encryptionkey@0"
I20260929 19:04:31.720258 17374 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708447806542-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708447806542-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708447806542-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:04:31.728678 17374 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:04:31.730269 17374 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:04:31.730365 17374 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:04:31.733805 17374 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34385
I20260929 19:04:31.733830 17441 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34385 every 8 connection(s)
I20260929 19:04:31.734117 17374 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708447806542-392-0/minicluster-data/master-0/data/info.pb
I20260929 19:04:31.734869 17442 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:04:31.736889 17442 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5eae6349988c4802981353bc907e10d5: Bootstrap starting.
I20260929 19:04:31.737509 17442 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5eae6349988c4802981353bc907e10d5: Neither blocks nor log segments found. Creating new log.
I20260929 19:04:31.737519 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUbnLHE/build/release/bin/kudu as pid 17374
I20260929 19:04:31.737609 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708447806542-392-0/minicluster-data/master-0/wal/instance
I20260929 19:04:31.737823 392 external_mini_cluster.cc:1428] Setting key 5e9254c251d29f9b36536bfd3b5e1f32
I20260929 19:04:31.737826 17442 log.cc:824] T 00000000000000000000000000000000 P 5eae6349988c4802981353bc907e10d5: Log is configured to *not* fsync() on all Append() calls
I20260929 19:04:31.738610 17442 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5eae6349988c4802981353bc907e10d5: No bootstrap required, opened a new log
I20260929 19:04:31.740857 17442 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5eae6349988c4802981353bc907e10d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5eae6349988c4802981353bc907e10d5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34385 } }
I20260929 19:04:31.741004 17442 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5eae6349988c4802981353bc907e10d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:04:31.741036 17442 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5eae6349988c4802981353bc907e10d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5eae6349988c4802981353bc907e10d5, State: Initialized, Role: FOLLOWER
I20260929 19:04:31.741145 17442 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5eae6349988c4802981353bc907e10d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5eae6349988c4802981353bc907e10d5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34385 } }
I20260929 19:04:31.741212 17442 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5eae6349988c4802981353bc907e10d5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 19:04:31.741240 17442 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5eae6349988c4802981353bc907e10d5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 19:04:31.741281 17442 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5eae6349988c4802981353bc907e10d5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:04:31.741891 17442 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5eae6349988c4802981353bc907e10d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5eae6349988c4802981353bc907e10d5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34385 } }
I20260929 19:04:31.741972 17442 leader_election.cc:302] T 00000000000000000000000000000000 P 5eae6349988c4802981353bc907e10d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5eae6349988c4802981353bc907e10d5; no voters:
I20260929 19:04:31.742156 17442 leader_election.cc:288] T 00000000000000000000000000000000 P 5eae6349988c4802981353bc907e10d5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 19:04:31.742197 17447 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5eae6349988c4802981353bc907e10d5 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:04:31.742360 17447 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5eae6349988c4802981353bc907e10d5 [term 1 LEADER]: Becoming Leader. State: Replica: 5eae6349988c4802981353bc907e10d5, State: Running, Role: LEADER
I20260929 19:04:31.742408 17442 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5eae6349988c4802981353bc907e10d5 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 19:04:31.742499 17447 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5eae6349988c4802981353bc907e10d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5eae6349988c4802981353bc907e10d5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34385 } }
I20260929 19:04:31.742789 17447 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5eae6349988c4802981353bc907e10d5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5eae6349988c4802981353bc907e10d5. Latest consensus state: current_term: 1 leader_uuid: "5eae6349988c4802981353bc907e10d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5eae6349988c4802981353bc907e10d5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34385 } } }
I20260929 19:04:31.742877 17447 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5eae6349988c4802981353bc907e10d5 [sys.catalog]: This master's current role is: LEADER
I20260929 19:04:31.742800 17448 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5eae6349988c4802981353bc907e10d5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5eae6349988c4802981353bc907e10d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5eae6349988c4802981353bc907e10d5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34385 } } }
I20260929 19:04:31.743049 17448 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5eae6349988c4802981353bc907e10d5 [sys.catalog]: This master's current role is: LEADER
I20260929 19:04:31.743091 17454 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 19:04:31.743574 17454 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 19:04:31.745203 17454 catalog_manager.cc:1347] Generated new cluster ID: 67b78668516c4127b6d55694c9fd8bd0
I20260929 19:04:31.745262 17454 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 29 19:04:31 dist-test-slave-lqkv krb5kdc[17361](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708671, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 19:04:31.749212 17454 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 19:04:31.749845 17454 catalog_manager.cc:1504] Loading token signing keys...
I20260929 19:04:31.750101 17446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:31.738250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60566 (local address 127.0.98.62:34385)
0929 19:04:31.738400 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0929 19:04:31.738414 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 19:04:31.738457 (+ 43us) server_negotiation.cc:407] Connection header received
0929 19:04:31.738934 (+ 477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:04:31.738940 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:04:31.738981 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:04:31.739081 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:04:31.739812 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:31.740674 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:31.741704 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:31.742253 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:31.742616 (+ 363us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:04:31.746466 (+ 3850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:04:31.746470 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:04:31.746470 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:04:31.746480 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:04:31.748372 (+ 1892us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:04:31.748760 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:04:31.748764 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:04:31.748782 (+ 18us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:04:31.748902 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:04:31.749100 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:04:31.749101 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:04:31.749102 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:04:31.749356 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:04:31.749526 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0929 19:04:31.749622 (+ 96us) server_negotiation.cc:299] Negotiation successful
0929 19:04:31.749910 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":145,"server-negotiator.queue_time_us":107,"thread_start_us":71,"threads_started":1}
I20260929 19:04:31.755649 17454 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5eae6349988c4802981353bc907e10d5: Generated new TSK 0
I20260929 19:04:31.755802 17454 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708447806542-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708447806542-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260929 19:04:31.783622 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUbnLHE/build/release/bin/kudu
/tmp/dist-test-taskUbnLHE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708447806542-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708447806542-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708447806542-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708447806542-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34385
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708447806542-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708447806542-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708447806542-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708447806542-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:04:31.858173 17468 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:04:31.858371 17468 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:04:31.858393 17468 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:04:31.859472 17468 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:04:31.859511 17468 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:04:31.859562 17468 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 19:04:31.860985 17468 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708447806542-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708447806542-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708447806542-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708447806542-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34385
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17468
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708447806542-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 18:09:13 UTC on ccycloud.quasar-hpkalf.root.comops.site
build id 4680913
I20260929 19:04:31.861217 17468 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:04:31.861430 17468 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:04:31.865986 17476 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:04:31.865991 17474 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:04:31.866000 17473 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:04:31.866113 17468 server_base.cc:1034] running on GCE node
I20260929 19:04:31.866384 17468 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:04:31.866564 17468 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:04:31.867753 17468 hybrid_clock.cc:648] HybridClock initialized: now 1790708671867738 us; error 36 us; skew 500 ppm
Sep 29 19:04:31 dist-test-slave-lqkv krb5kdc[17361](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708671, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:04:31.870069 17468 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 19:04:31.870457 17468 webserver.cc:460] Webserver started at http://127.0.98.1:38667/ using document root <none> and password file <none>
I20260929 19:04:31.870656 17468 fs_manager.cc:359] Metadata directory not provided
I20260929 19:04:31.870723 17468 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:04:31.870868 17468 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:04:31.871910 17468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708447806542-392-0/minicluster-data/ts-0/data/instance:
uuid: "4641ddfd69ef42d09bfad49bbdddaa4a"
format_stamp: "Formatted at 2026-09-29 19:04:31 on dist-test-slave-lqkv"
server_key: "c99b103e446c28a0a20ad83766253b6e"
server_key_iv: "5817d5f7920e52e9defaec9b905e595a"
server_key_version: "encryptionkey@0"
I20260929 19:04:31.872248 17468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708447806542-392-0/minicluster-data/ts-0/wal/instance:
uuid: "4641ddfd69ef42d09bfad49bbdddaa4a"
format_stamp: "Formatted at 2026-09-29 19:04:31 on dist-test-slave-lqkv"
server_key: "c99b103e446c28a0a20ad83766253b6e"
server_key_iv: "5817d5f7920e52e9defaec9b905e595a"
server_key_version: "encryptionkey@0"
I20260929 19:04:31.873523 17468 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 19:04:31.874154 17483 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:04:31.874310 17468 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260929 19:04:31.874399 17468 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708447806542-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708447806542-392-0/minicluster-data/ts-0/wal
uuid: "4641ddfd69ef42d09bfad49bbdddaa4a"
format_stamp: "Formatted at 2026-09-29 19:04:31 on dist-test-slave-lqkv"
server_key: "c99b103e446c28a0a20ad83766253b6e"
server_key_iv: "5817d5f7920e52e9defaec9b905e595a"
server_key_version: "encryptionkey@0"
I20260929 19:04:31.874475 17468 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708447806542-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708447806542-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708447806542-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:04:31.898370 17468 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:04:31.900133 17468 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:04:31.900295 17468 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:04:31.900508 17468 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:04:31.900851 17468 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:04:31.900920 17468 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:04:31.900959 17468 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:04:31.900987 17468 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:04:31.906634 17468 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45117
I20260929 19:04:31.906669 17596 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45117 every 8 connection(s)
I20260929 19:04:31.906953 17468 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708447806542-392-0/minicluster-data/ts-0/data/info.pb
I20260929 19:04:31.907459 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUbnLHE/build/release/bin/kudu as pid 17468
I20260929 19:04:31.907565 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708447806542-392-0/minicluster-data/ts-0/wal/instance
I20260929 19:04:31.907713 392 external_mini_cluster.cc:1428] Setting key e3b13a146e46028a8820f21d4c0f1144
Sep 29 19:04:31 dist-test-slave-lqkv krb5kdc[17361](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708671, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260929 19:04:31.917357 17446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:31.908240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42249 (local address 127.0.98.62:34385)
0929 19:04:31.908285 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0929 19:04:31.908287 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:04:31.908576 (+ 289us) server_negotiation.cc:407] Connection header received
0929 19:04:31.909478 (+ 902us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:04:31.909480 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:04:31.909491 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:04:31.909555 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:04:31.910307 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:31.910947 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:31.912025 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:31.912382 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:31.912452 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:04:31.914716 (+ 2264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:04:31.914717 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:04:31.914717 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:04:31.914723 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:04:31.916063 (+ 1340us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:04:31.916476 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:04:31.916477 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:04:31.916478 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:04:31.916535 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:04:31.916752 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:04:31.916753 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:04:31.916753 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:04:31.916906 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:04:31.916958 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0929 19:04:31.917223 (+ 265us) server_negotiation.cc:299] Negotiation successful
0929 19:04:31.917295 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260929 19:04:31.917532 17599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:31.908214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34385 (local address 127.0.98.1:42249)
0929 19:04:31.908432 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0929 19:04:31.908487 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0929 19:04:31.909385 (+ 898us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:04:31.909643 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:04:31.909647 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:04:31.909982 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:04:31.910219 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:04:31.910224 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:04:31.911030 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:04:31.911030 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:04:31.911934 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:04:31.911936 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:04:31.912468 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:04:31.912469 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:04:31.912597 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:04:31.913116 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:04:31.913132 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:04:31.914633 (+ 1501us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:04:31.916134 (+ 1501us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:04:31.916140 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:04:31.916141 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:04:31.916405 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:04:31.916595 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:04:31.916595 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:04:31.916595 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:04:31.916684 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:04:31.916961 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:04:31.916966 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:04:31.917067 (+ 101us) client_negotiation.cc:769] Sending connection context
0929 19:04:31.917194 (+ 127us) client_negotiation.cc:240] Negotiation successful
0929 19:04:31.917340 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":104,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260929 19:04:31.918047 17597 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34385
I20260929 19:04:31.918144 17597 heartbeater.cc:464] Registering TS with master...
I20260929 19:04:31.918303 17597 heartbeater.cc:511] Master 127.0.98.62:34385 requested a full tablet report, sending...
I20260929 19:04:31.918807 17406 ts_manager.cc:194] Registered new tserver with Master: 4641ddfd69ef42d09bfad49bbdddaa4a (127.0.98.1:45117)
I20260929 19:04:31.919684 17406 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:42249
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708447806542-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708447806542-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260929 19:04:31.931124 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUbnLHE/build/release/bin/kudu
/tmp/dist-test-taskUbnLHE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708447806542-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708447806542-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708447806542-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708447806542-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34385
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708447806542-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708447806542-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708447806542-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708447806542-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:04:32.004163 17602 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:04:32.004366 17602 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:04:32.004420 17602 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:04:32.005488 17602 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:04:32.005522 17602 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:04:32.005545 17602 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 19:04:32.006914 17602 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708447806542-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708447806542-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708447806542-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708447806542-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34385
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17602
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708447806542-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 18:09:13 UTC on ccycloud.quasar-hpkalf.root.comops.site
build id 4680913
I20260929 19:04:32.007073 17602 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:04:32.007265 17602 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:04:32.011628 17608 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:04:32.011667 17602 server_base.cc:1034] running on GCE node
W20260929 19:04:32.011710 17607 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:04:32.011636 17610 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:04:32.012015 17602 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:04:32.012243 17602 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:04:32.013392 17602 hybrid_clock.cc:648] HybridClock initialized: now 1790708672013384 us; error 46 us; skew 500 ppm
Sep 29 19:04:32 dist-test-slave-lqkv krb5kdc[17361](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708672, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:04:32.015666 17602 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 19:04:32.016077 17602 webserver.cc:460] Webserver started at http://127.0.98.2:37035/ using document root <none> and password file <none>
I20260929 19:04:32.016286 17602 fs_manager.cc:359] Metadata directory not provided
I20260929 19:04:32.016360 17602 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:04:32.016505 17602 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:04:32.017513 17602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708447806542-392-0/minicluster-data/ts-1/data/instance:
uuid: "218aa16cad18435d8f9dec9ba96839ca"
format_stamp: "Formatted at 2026-09-29 19:04:32 on dist-test-slave-lqkv"
server_key: "203e20a84e86e32544966047aa28023c"
server_key_iv: "553f17f136099e24a1de35106dfaa419"
server_key_version: "encryptionkey@0"
I20260929 19:04:32.017841 17602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708447806542-392-0/minicluster-data/ts-1/wal/instance:
uuid: "218aa16cad18435d8f9dec9ba96839ca"
format_stamp: "Formatted at 2026-09-29 19:04:32 on dist-test-slave-lqkv"
server_key: "203e20a84e86e32544966047aa28023c"
server_key_iv: "553f17f136099e24a1de35106dfaa419"
server_key_version: "encryptionkey@0"
I20260929 19:04:32.019011 17602 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260929 19:04:32.019649 17617 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:04:32.019793 17602 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 19:04:32.019877 17602 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708447806542-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708447806542-392-0/minicluster-data/ts-1/wal
uuid: "218aa16cad18435d8f9dec9ba96839ca"
format_stamp: "Formatted at 2026-09-29 19:04:32 on dist-test-slave-lqkv"
server_key: "203e20a84e86e32544966047aa28023c"
server_key_iv: "553f17f136099e24a1de35106dfaa419"
server_key_version: "encryptionkey@0"
I20260929 19:04:32.019948 17602 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708447806542-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708447806542-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708447806542-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:04:32.031301 17602 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:04:32.032900 17602 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:04:32.033039 17602 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:04:32.033217 17602 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:04:32.033473 17602 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:04:32.033530 17602 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:04:32.033564 17602 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:04:32.033589 17602 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:04:32.038949 17602 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38101
I20260929 19:04:32.038990 17730 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38101 every 8 connection(s)
I20260929 19:04:32.039237 17602 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708447806542-392-0/minicluster-data/ts-1/data/info.pb
Sep 29 19:04:32 dist-test-slave-lqkv krb5kdc[17361](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708672, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 19:04:32.044955 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUbnLHE/build/release/bin/kudu as pid 17602
I20260929 19:04:32.045076 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708447806542-392-0/minicluster-data/ts-1/wal/instance
I20260929 19:04:32.045248 392 external_mini_cluster.cc:1428] Setting key 0a140a8264acc90f6ebc4a6d80022816
I20260929 19:04:32.047967 17733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:32.039816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34385 (local address 127.0.98.2:38509)
0929 19:04:32.039950 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0929 19:04:32.039985 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0929 19:04:32.040769 (+ 784us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:04:32.041024 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:04:32.041027 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:04:32.041238 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:04:32.041402 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:04:32.041406 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:04:32.041974 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:04:32.041975 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:04:32.042756 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:04:32.042758 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:04:32.043080 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:04:32.043081 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:04:32.043167 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:04:32.043549 (+ 382us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:04:32.043555 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:04:32.044969 (+ 1414us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:04:32.046646 (+ 1677us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:04:32.046653 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:04:32.046654 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:04:32.046960 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:04:32.047203 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:04:32.047204 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:04:32.047204 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:04:32.047295 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:04:32.047566 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:04:32.047568 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:04:32.047661 (+ 93us) client_negotiation.cc:769] Sending connection context
0929 19:04:32.047731 (+ 70us) client_negotiation.cc:240] Negotiation successful
0929 19:04:32.047835 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":69,"threads_started":1}
I20260929 19:04:32.047994 17446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:32.039844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38509 (local address 127.0.98.62:34385)
0929 19:04:32.039908 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0929 19:04:32.039910 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:04:32.040066 (+ 156us) server_negotiation.cc:407] Connection header received
0929 19:04:32.040913 (+ 847us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:04:32.040914 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:04:32.040922 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:04:32.040976 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:04:32.041466 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:32.041908 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:32.042819 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:32.043001 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:32.043074 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:04:32.045105 (+ 2031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:04:32.045107 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:04:32.045107 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:04:32.045113 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:04:32.046569 (+ 1456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:04:32.047044 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:04:32.047046 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:04:32.047046 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:04:32.047119 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:04:32.047365 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:04:32.047366 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:04:32.047367 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:04:32.047502 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:04:32.047579 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0929 19:04:32.047775 (+ 196us) server_negotiation.cc:299] Negotiation successful
0929 19:04:32.047867 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 19:04:32.048692 17731 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34385
I20260929 19:04:32.048791 17731 heartbeater.cc:464] Registering TS with master...
I20260929 19:04:32.048987 17731 heartbeater.cc:511] Master 127.0.98.62:34385 requested a full tablet report, sending...
I20260929 19:04:32.049314 17406 ts_manager.cc:194] Registered new tserver with Master: 218aa16cad18435d8f9dec9ba96839ca (127.0.98.2:38101)
I20260929 19:04:32.049642 17406 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:38509
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708447806542-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708447806542-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260929 19:04:32.067142 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUbnLHE/build/release/bin/kudu
/tmp/dist-test-taskUbnLHE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708447806542-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708447806542-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708447806542-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708447806542-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34385
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708447806542-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708447806542-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708447806542-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708447806542-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:04:32.138865 17736 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:04:32.139055 17736 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:04:32.139088 17736 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:04:32.140175 17736 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:04:32.140239 17736 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:04:32.140290 17736 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 19:04:32.141640 17736 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708447806542-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708447806542-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708447806542-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708447806542-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34385
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17736
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708447806542-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 18:09:13 UTC on ccycloud.quasar-hpkalf.root.comops.site
build id 4680913
I20260929 19:04:32.141906 17736 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:04:32.142129 17736 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 19:04:32.146701 17736 server_base.cc:1034] running on GCE node
W20260929 19:04:32.146713 17742 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:04:32.146713 17741 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:04:32.146891 17744 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:04:32.147120 17736 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:04:32.147359 17736 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:04:32.148489 17736 hybrid_clock.cc:648] HybridClock initialized: now 1790708672148478 us; error 28 us; skew 500 ppm
Sep 29 19:04:32 dist-test-slave-lqkv krb5kdc[17361](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708672, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:04:32.150830 17736 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 19:04:32.151199 17736 webserver.cc:460] Webserver started at http://127.0.98.3:36771/ using document root <none> and password file <none>
I20260929 19:04:32.151350 17736 fs_manager.cc:359] Metadata directory not provided
I20260929 19:04:32.151384 17736 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:04:32.151456 17736 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:04:32.152333 17736 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708447806542-392-0/minicluster-data/ts-2/data/instance:
uuid: "f41e17f9fc744806a4333399545da3e1"
format_stamp: "Formatted at 2026-09-29 19:04:32 on dist-test-slave-lqkv"
server_key: "0c3e5429d32459b915cdc3645f813001"
server_key_iv: "19acd3903f8eb51f623fefac931318c5"
server_key_version: "encryptionkey@0"
I20260929 19:04:32.152599 17736 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708447806542-392-0/minicluster-data/ts-2/wal/instance:
uuid: "f41e17f9fc744806a4333399545da3e1"
format_stamp: "Formatted at 2026-09-29 19:04:32 on dist-test-slave-lqkv"
server_key: "0c3e5429d32459b915cdc3645f813001"
server_key_iv: "19acd3903f8eb51f623fefac931318c5"
server_key_version: "encryptionkey@0"
I20260929 19:04:32.153753 17736 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260929 19:04:32.154456 17751 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:04:32.154603 17736 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 19:04:32.154690 17736 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708447806542-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708447806542-392-0/minicluster-data/ts-2/wal
uuid: "f41e17f9fc744806a4333399545da3e1"
format_stamp: "Formatted at 2026-09-29 19:04:32 on dist-test-slave-lqkv"
server_key: "0c3e5429d32459b915cdc3645f813001"
server_key_iv: "19acd3903f8eb51f623fefac931318c5"
server_key_version: "encryptionkey@0"
I20260929 19:04:32.154762 17736 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708447806542-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708447806542-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708447806542-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:04:32.181981 17736 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:04:32.183602 17736 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:04:32.183751 17736 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:04:32.183952 17736 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:04:32.184229 17736 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:04:32.184283 17736 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:04:32.184333 17736 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:04:32.184370 17736 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:04:32.189929 17736 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38303
I20260929 19:04:32.189961 17864 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38303 every 8 connection(s)
I20260929 19:04:32.190235 17736 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708447806542-392-0/minicluster-data/ts-2/data/info.pb
I20260929 19:04:32.191133 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUbnLHE/build/release/bin/kudu as pid 17736
I20260929 19:04:32.191242 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708447806542-392-0/minicluster-data/ts-2/wal/instance
I20260929 19:04:32.191411 392 external_mini_cluster.cc:1428] Setting key 26147e03f90e73933fe7e94e75ab1a2b
Sep 29 19:04:32 dist-test-slave-lqkv krb5kdc[17361](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708672, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 19:04:32.198275 17446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:32.190886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52233 (local address 127.0.98.62:34385)
0929 19:04:32.190954 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0929 19:04:32.190957 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 19:04:32.191159 (+ 202us) server_negotiation.cc:407] Connection header received
0929 19:04:32.192310 (+ 1151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:04:32.192311 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:04:32.192345 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:04:32.192400 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:04:32.192950 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:32.193380 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:32.194148 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:32.194295 (+ 147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:32.194342 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:04:32.196125 (+ 1783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:04:32.196126 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:04:32.196126 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:04:32.196133 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:04:32.197288 (+ 1155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:04:32.197613 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:04:32.197614 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:04:32.197614 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:04:32.197657 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:04:32.197842 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:04:32.197843 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:04:32.197843 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:04:32.197928 (+ 85us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:04:32.197973 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0929 19:04:32.198168 (+ 195us) server_negotiation.cc:299] Negotiation successful
0929 19:04:32.198218 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260929 19:04:32.198406 17867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:32.190843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34385 (local address 127.0.98.3:52233)
0929 19:04:32.191021 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0929 19:04:32.191064 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 19:04:32.192217 (+ 1153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:04:32.192453 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:04:32.192455 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:04:32.192664 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:04:32.192867 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:04:32.192871 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:04:32.193440 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:04:32.193441 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:04:32.194088 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:04:32.194089 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:04:32.194357 (+ 268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:04:32.194358 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:04:32.194440 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:04:32.194852 (+ 412us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:04:32.194859 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:04:32.196047 (+ 1188us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:04:32.197359 (+ 1312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:04:32.197363 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:04:32.197365 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:04:32.197552 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:04:32.197708 (+ 156us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:04:32.197709 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:04:32.197709 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:04:32.197793 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:04:32.197976 (+ 183us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:04:32.197980 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:04:32.198071 (+ 91us) client_negotiation.cc:769] Sending connection context
0929 19:04:32.198143 (+ 72us) client_negotiation.cc:240] Negotiation successful
0929 19:04:32.198241 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":77,"threads_started":1}
I20260929 19:04:32.198773 17865 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34385
I20260929 19:04:32.198836 17865 heartbeater.cc:464] Registering TS with master...
I20260929 19:04:32.198942 17865 heartbeater.cc:511] Master 127.0.98.62:34385 requested a full tablet report, sending...
I20260929 19:04:32.199213 17406 ts_manager.cc:194] Registered new tserver with Master: f41e17f9fc744806a4333399545da3e1 (127.0.98.3:38303)
I20260929 19:04:32.199545 17406 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:52233
I20260929 19:04:32.202991 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 19:04:32.209300 17446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:32.204129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60578 (local address 127.0.98.62:34385)
0929 19:04:32.204159 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0929 19:04:32.204161 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:04:32.204292 (+ 131us) server_negotiation.cc:407] Connection header received
0929 19:04:32.204329 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:04:32.204330 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:04:32.204336 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:04:32.204380 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:04:32.204893 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:32.205268 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:32.205939 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:32.206100 (+ 161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:32.206145 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:04:32.207108 (+ 963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:04:32.207109 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:04:32.207109 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:04:32.207114 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:04:32.208423 (+ 1309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:04:32.208739 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:04:32.208740 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:04:32.208740 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:04:32.208780 (+ 40us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:04:32.208967 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:04:32.208967 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:04:32.208967 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:04:32.209071 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:04:32.209135 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0929 19:04:32.209209 (+ 74us) server_negotiation.cc:299] Negotiation successful
0929 19:04:32.209255 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260929 19:04:32.210731 17406 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60578:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 19:04:32.212100 17406 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 19:04:32.217522 17873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:32.213964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38303 (local address 127.0.0.1:38606)
0929 19:04:32.214314 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0929 19:04:32.214324 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 19:04:32.214396 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:04:32.214599 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:04:32.214602 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:04:32.214603 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:04:32.214649 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:04:32.214651 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:04:32.215485 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:04:32.215486 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:04:32.216493 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:04:32.216495 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:04:32.217267 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:04:32.217268 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:04:32.217337 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:04:32.217349 (+ 12us) client_negotiation.cc:769] Sending connection context
0929 19:04:32.217415 (+ 66us) client_negotiation.cc:240] Negotiation successful
0929 19:04:32.217459 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":78,"threads_started":1}
I20260929 19:04:32.217530 17877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:32.214188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38606 (local address 127.0.98.3:38303)
0929 19:04:32.214307 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0929 19:04:32.214311 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 19:04:32.214383 (+ 72us) server_negotiation.cc:407] Connection header received
0929 19:04:32.214458 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:04:32.214460 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:04:32.214474 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:04:32.214544 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:04:32.214717 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:32.215390 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:32.216563 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:32.217192 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:32.217256 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:04:32.217302 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0929 19:04:32.217434 (+ 132us) server_negotiation.cc:299] Negotiation successful
0929 19:04:32.217464 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93,"thread_start_us":63,"threads_started":1}
I20260929 19:04:32.218103 17874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:32.213931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45117 (local address 127.0.0.1:39664)
0929 19:04:32.214092 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0929 19:04:32.214127 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0929 19:04:32.214186 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:04:32.214599 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:04:32.214601 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:04:32.214607 (+ 6us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:04:32.214656 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:04:32.214657 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:04:32.215955 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:04:32.215955 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:04:32.216999 (+ 1044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:04:32.217001 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:04:32.217887 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:04:32.217888 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:04:32.217938 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:04:32.217941 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 19:04:32.218012 (+ 71us) client_negotiation.cc:240] Negotiation successful
0929 19:04:32.218045 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":62,"threads_started":1}
I20260929 19:04:32.218114 17875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:32.214027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39664 (local address 127.0.98.1:45117)
0929 19:04:32.214421 (+ 394us) server_negotiation.cc:206] Beginning negotiation
0929 19:04:32.214424 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 19:04:32.214432 (+ 8us) server_negotiation.cc:407] Connection header received
0929 19:04:32.214472 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:04:32.214474 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:04:32.214484 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:04:32.214788 (+ 304us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:04:32.214879 (+ 91us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:32.215846 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:32.217077 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:32.217803 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:32.217873 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:04:32.217926 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0929 19:04:32.218031 (+ 105us) server_negotiation.cc:299] Negotiation successful
0929 19:04:32.218061 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":76,"threads_started":1}
I20260929 19:04:32.218266 17799 tablet_service.cc:1467] Processing CreateTablet for tablet a81592216cd047d6bf03992a97afac0d (DEFAULT_TABLE table=test-table [id=b6f086f9e7d64c5c9ffb0416786351ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:04:32.218525 17799 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a81592216cd047d6bf03992a97afac0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:04:32.220917 17879 tablet_bootstrap.cc:492] T a81592216cd047d6bf03992a97afac0d P f41e17f9fc744806a4333399545da3e1: Bootstrap starting.
I20260929 19:04:32.221216 17531 tablet_service.cc:1467] Processing CreateTablet for tablet a81592216cd047d6bf03992a97afac0d (DEFAULT_TABLE table=test-table [id=b6f086f9e7d64c5c9ffb0416786351ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:04:32.221477 17531 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a81592216cd047d6bf03992a97afac0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:04:32.221482 17879 tablet_bootstrap.cc:654] T a81592216cd047d6bf03992a97afac0d P f41e17f9fc744806a4333399545da3e1: Neither blocks nor log segments found. Creating new log.
I20260929 19:04:32.222093 17879 log.cc:824] T a81592216cd047d6bf03992a97afac0d P f41e17f9fc744806a4333399545da3e1: Log is configured to *not* fsync() on all Append() calls
I20260929 19:04:32.222836 17879 tablet_bootstrap.cc:492] T a81592216cd047d6bf03992a97afac0d P f41e17f9fc744806a4333399545da3e1: No bootstrap required, opened a new log
I20260929 19:04:32.222965 17879 ts_tablet_manager.cc:1397] T a81592216cd047d6bf03992a97afac0d P f41e17f9fc744806a4333399545da3e1: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 19:04:32.223773 17876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:32.214121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38101 (local address 127.0.0.1:39198)
0929 19:04:32.214476 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0929 19:04:32.214484 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0929 19:04:32.214530 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:04:32.214674 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:04:32.214675 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:04:32.214677 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:04:32.214711 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:04:32.214712 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:04:32.217068 (+ 2356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:04:32.217068 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:04:32.222548 (+ 5480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:04:32.222551 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:04:32.223507 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:04:32.223508 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:04:32.223571 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:04:32.223576 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 19:04:32.223662 (+ 86us) client_negotiation.cc:240] Negotiation successful
0929 19:04:32.223699 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":59,"threads_started":1}
I20260929 19:04:32.223803 17878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:32.214221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39198 (local address 127.0.98.2:38101)
0929 19:04:32.214379 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0929 19:04:32.214383 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 19:04:32.214548 (+ 165us) server_negotiation.cc:407] Connection header received
0929 19:04:32.214588 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:04:32.214590 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:04:32.214603 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:04:32.214669 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:04:32.214806 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:32.215835 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:32.222634 (+ 6799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:32.223438 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:32.223504 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:04:32.223555 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0929 19:04:32.223684 (+ 129us) server_negotiation.cc:299] Negotiation successful
0929 19:04:32.223728 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":71,"threads_started":1}
I20260929 19:04:32.224668 17881 tablet_bootstrap.cc:492] T a81592216cd047d6bf03992a97afac0d P 4641ddfd69ef42d09bfad49bbdddaa4a: Bootstrap starting.
I20260929 19:04:32.224622 17665 tablet_service.cc:1467] Processing CreateTablet for tablet a81592216cd047d6bf03992a97afac0d (DEFAULT_TABLE table=test-table [id=b6f086f9e7d64c5c9ffb0416786351ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:04:32.224949 17665 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a81592216cd047d6bf03992a97afac0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:04:32.224798 17879 raft_consensus.cc:348] T a81592216cd047d6bf03992a97afac0d P f41e17f9fc744806a4333399545da3e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4641ddfd69ef42d09bfad49bbdddaa4a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45117 } } peers { permanent_uuid: "218aa16cad18435d8f9dec9ba96839ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38101 } } peers { permanent_uuid: "f41e17f9fc744806a4333399545da3e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38303 } }
I20260929 19:04:32.225003 17879 raft_consensus.cc:374] T a81592216cd047d6bf03992a97afac0d P f41e17f9fc744806a4333399545da3e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:04:32.225070 17879 raft_consensus.cc:729] T a81592216cd047d6bf03992a97afac0d P f41e17f9fc744806a4333399545da3e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f41e17f9fc744806a4333399545da3e1, State: Initialized, Role: FOLLOWER
I20260929 19:04:32.225200 17879 consensus_queue.cc:260] T a81592216cd047d6bf03992a97afac0d P f41e17f9fc744806a4333399545da3e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4641ddfd69ef42d09bfad49bbdddaa4a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45117 } } peers { permanent_uuid: "218aa16cad18435d8f9dec9ba96839ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38101 } } peers { permanent_uuid: "f41e17f9fc744806a4333399545da3e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38303 } }
I20260929 19:04:32.225319 17881 tablet_bootstrap.cc:654] T a81592216cd047d6bf03992a97afac0d P 4641ddfd69ef42d09bfad49bbdddaa4a: Neither blocks nor log segments found. Creating new log.
I20260929 19:04:32.225488 17879 ts_tablet_manager.cc:1428] T a81592216cd047d6bf03992a97afac0d P f41e17f9fc744806a4333399545da3e1: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260929 19:04:32.225601 17865 heartbeater.cc:503] Master 127.0.98.62:34385 was elected leader, sending a full tablet report...
I20260929 19:04:32.225893 17881 log.cc:824] T a81592216cd047d6bf03992a97afac0d P 4641ddfd69ef42d09bfad49bbdddaa4a: Log is configured to *not* fsync() on all Append() calls
I20260929 19:04:32.226709 17881 tablet_bootstrap.cc:492] T a81592216cd047d6bf03992a97afac0d P 4641ddfd69ef42d09bfad49bbdddaa4a: No bootstrap required, opened a new log
I20260929 19:04:32.226792 17881 ts_tablet_manager.cc:1397] T a81592216cd047d6bf03992a97afac0d P 4641ddfd69ef42d09bfad49bbdddaa4a: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260929 19:04:32.227843 17884 tablet_bootstrap.cc:492] T a81592216cd047d6bf03992a97afac0d P 218aa16cad18435d8f9dec9ba96839ca: Bootstrap starting.
I20260929 19:04:32.228451 17884 tablet_bootstrap.cc:654] T a81592216cd047d6bf03992a97afac0d P 218aa16cad18435d8f9dec9ba96839ca: Neither blocks nor log segments found. Creating new log.
I20260929 19:04:32.228343 17881 raft_consensus.cc:348] T a81592216cd047d6bf03992a97afac0d P 4641ddfd69ef42d09bfad49bbdddaa4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4641ddfd69ef42d09bfad49bbdddaa4a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45117 } } peers { permanent_uuid: "218aa16cad18435d8f9dec9ba96839ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38101 } } peers { permanent_uuid: "f41e17f9fc744806a4333399545da3e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38303 } }
I20260929 19:04:32.228504 17881 raft_consensus.cc:374] T a81592216cd047d6bf03992a97afac0d P 4641ddfd69ef42d09bfad49bbdddaa4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:04:32.228549 17881 raft_consensus.cc:729] T a81592216cd047d6bf03992a97afac0d P 4641ddfd69ef42d09bfad49bbdddaa4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4641ddfd69ef42d09bfad49bbdddaa4a, State: Initialized, Role: FOLLOWER
I20260929 19:04:32.228653 17881 consensus_queue.cc:260] T a81592216cd047d6bf03992a97afac0d P 4641ddfd69ef42d09bfad49bbdddaa4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4641ddfd69ef42d09bfad49bbdddaa4a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45117 } } peers { permanent_uuid: "218aa16cad18435d8f9dec9ba96839ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38101 } } peers { permanent_uuid: "f41e17f9fc744806a4333399545da3e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38303 } }
I20260929 19:04:32.228937 17597 heartbeater.cc:503] Master 127.0.98.62:34385 was elected leader, sending a full tablet report...
I20260929 19:04:32.228964 17884 log.cc:824] T a81592216cd047d6bf03992a97afac0d P 218aa16cad18435d8f9dec9ba96839ca: Log is configured to *not* fsync() on all Append() calls
I20260929 19:04:32.228950 17881 ts_tablet_manager.cc:1428] T a81592216cd047d6bf03992a97afac0d P 4641ddfd69ef42d09bfad49bbdddaa4a: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260929 19:04:32.229655 17884 tablet_bootstrap.cc:492] T a81592216cd047d6bf03992a97afac0d P 218aa16cad18435d8f9dec9ba96839ca: No bootstrap required, opened a new log
I20260929 19:04:32.229718 17884 ts_tablet_manager.cc:1397] T a81592216cd047d6bf03992a97afac0d P 218aa16cad18435d8f9dec9ba96839ca: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260929 19:04:32.230933 17884 raft_consensus.cc:348] T a81592216cd047d6bf03992a97afac0d P 218aa16cad18435d8f9dec9ba96839ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4641ddfd69ef42d09bfad49bbdddaa4a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45117 } } peers { permanent_uuid: "218aa16cad18435d8f9dec9ba96839ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38101 } } peers { permanent_uuid: "f41e17f9fc744806a4333399545da3e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38303 } }
I20260929 19:04:32.231076 17884 raft_consensus.cc:374] T a81592216cd047d6bf03992a97afac0d P 218aa16cad18435d8f9dec9ba96839ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:04:32.231109 17884 raft_consensus.cc:729] T a81592216cd047d6bf03992a97afac0d P 218aa16cad18435d8f9dec9ba96839ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 218aa16cad18435d8f9dec9ba96839ca, State: Initialized, Role: FOLLOWER
I20260929 19:04:32.231211 17884 consensus_queue.cc:260] T a81592216cd047d6bf03992a97afac0d P 218aa16cad18435d8f9dec9ba96839ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4641ddfd69ef42d09bfad49bbdddaa4a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45117 } } peers { permanent_uuid: "218aa16cad18435d8f9dec9ba96839ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38101 } } peers { permanent_uuid: "f41e17f9fc744806a4333399545da3e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38303 } }
I20260929 19:04:32.231405 17884 ts_tablet_manager.cc:1428] T a81592216cd047d6bf03992a97afac0d P 218aa16cad18435d8f9dec9ba96839ca: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260929 19:04:32.231436 17731 heartbeater.cc:503] Master 127.0.98.62:34385 was elected leader, sending a full tablet report...
W20260929 19:04:32.289690 17732 tablet.cc:2369] T a81592216cd047d6bf03992a97afac0d P 218aa16cad18435d8f9dec9ba96839ca: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 19:04:32.295076 17885 raft_consensus.cc:482] T a81592216cd047d6bf03992a97afac0d P 4641ddfd69ef42d09bfad49bbdddaa4a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 19:04:32.295188 17885 raft_consensus.cc:504] T a81592216cd047d6bf03992a97afac0d P 4641ddfd69ef42d09bfad49bbdddaa4a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4641ddfd69ef42d09bfad49bbdddaa4a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45117 } } peers { permanent_uuid: "218aa16cad18435d8f9dec9ba96839ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38101 } } peers { permanent_uuid: "f41e17f9fc744806a4333399545da3e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38303 } }
I20260929 19:04:32.295527 17885 leader_election.cc:288] T a81592216cd047d6bf03992a97afac0d P 4641ddfd69ef42d09bfad49bbdddaa4a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 218aa16cad18435d8f9dec9ba96839ca (127.0.98.2:38101), f41e17f9fc744806a4333399545da3e1 (127.0.98.3:38303)
I20260929 19:04:32.299193 17877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:32.295753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54191 (local address 127.0.98.3:38303)
0929 19:04:32.295825 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0929 19:04:32.295827 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:04:32.295836 (+ 9us) server_negotiation.cc:407] Connection header received
0929 19:04:32.295871 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:04:32.295873 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:04:32.295882 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:04:32.295933 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:04:32.296133 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:32.296907 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:32.298277 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:32.298833 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:32.298910 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:04:32.298920 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0929 19:04:32.299091 (+ 171us) server_negotiation.cc:299] Negotiation successful
0929 19:04:32.299129 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 19:04:32.299193 17878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:32.295760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54077 (local address 127.0.98.2:38101)
0929 19:04:32.295799 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0929 19:04:32.295800 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 19:04:32.296075 (+ 275us) server_negotiation.cc:407] Connection header received
0929 19:04:32.296121 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:04:32.296123 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:04:32.296135 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:04:32.296191 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:04:32.296295 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:32.296986 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:32.298276 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:32.298833 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:32.298910 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:04:32.298920 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0929 19:04:32.299091 (+ 171us) server_negotiation.cc:299] Negotiation successful
0929 19:04:32.299129 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260929 19:04:32.299221 17599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:32.295681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38303 (local address 127.0.98.1:54191)
0929 19:04:32.295719 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0929 19:04:32.295730 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 19:04:32.295765 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:04:32.295961 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:04:32.295962 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:04:32.295964 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:04:32.296054 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:04:32.296056 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:04:32.296979 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:04:32.296980 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:04:32.298185 (+ 1205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:04:32.298187 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:04:32.298925 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:04:32.298926 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:04:32.299012 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:04:32.299016 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 19:04:32.299072 (+ 56us) client_negotiation.cc:240] Negotiation successful
0929 19:04:32.299124 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20,"mutex_wait_us":42}
I20260929 19:04:32.299227 17888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:32.295681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38101 (local address 127.0.98.1:54077)
0929 19:04:32.295995 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0929 19:04:32.296005 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 19:04:32.296067 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:04:32.296202 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:04:32.296203 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:04:32.296204 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:04:32.296250 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:04:32.296251 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:04:32.297058 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:04:32.297059 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:04:32.298185 (+ 1126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:04:32.298187 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:04:32.298926 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:04:32.298926 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:04:32.299012 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:04:32.299016 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 19:04:32.299072 (+ 56us) client_negotiation.cc:240] Negotiation successful
0929 19:04:32.299124 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"mutex_wait_us":169,"thread_start_us":91,"threads_started":1}
I20260929 19:04:32.299646 17819 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a81592216cd047d6bf03992a97afac0d" candidate_uuid: "4641ddfd69ef42d09bfad49bbdddaa4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f41e17f9fc744806a4333399545da3e1" is_pre_election: true
I20260929 19:04:32.299651 17685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a81592216cd047d6bf03992a97afac0d" candidate_uuid: "4641ddfd69ef42d09bfad49bbdddaa4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "218aa16cad18435d8f9dec9ba96839ca" is_pre_election: true
I20260929 19:04:32.299760 17819 raft_consensus.cc:2454] T a81592216cd047d6bf03992a97afac0d P f41e17f9fc744806a4333399545da3e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4641ddfd69ef42d09bfad49bbdddaa4a in term 0.
I20260929 19:04:32.299790 17685 raft_consensus.cc:2454] T a81592216cd047d6bf03992a97afac0d P 218aa16cad18435d8f9dec9ba96839ca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4641ddfd69ef42d09bfad49bbdddaa4a in term 0.
I20260929 19:04:32.300055 17487 leader_election.cc:302] T a81592216cd047d6bf03992a97afac0d P 4641ddfd69ef42d09bfad49bbdddaa4a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 218aa16cad18435d8f9dec9ba96839ca, 4641ddfd69ef42d09bfad49bbdddaa4a; no voters:
I20260929 19:04:32.300201 17885 raft_consensus.cc:2781] T a81592216cd047d6bf03992a97afac0d P 4641ddfd69ef42d09bfad49bbdddaa4a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 19:04:32.300243 17885 raft_consensus.cc:482] T a81592216cd047d6bf03992a97afac0d P 4641ddfd69ef42d09bfad49bbdddaa4a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 19:04:32.300256 17885 raft_consensus.cc:3037] T a81592216cd047d6bf03992a97afac0d P 4641ddfd69ef42d09bfad49bbdddaa4a [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:04:32.300830 17885 raft_consensus.cc:504] T a81592216cd047d6bf03992a97afac0d P 4641ddfd69ef42d09bfad49bbdddaa4a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4641ddfd69ef42d09bfad49bbdddaa4a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45117 } } peers { permanent_uuid: "218aa16cad18435d8f9dec9ba96839ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38101 } } peers { permanent_uuid: "f41e17f9fc744806a4333399545da3e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38303 } }
I20260929 19:04:32.300951 17885 leader_election.cc:288] T a81592216cd047d6bf03992a97afac0d P 4641ddfd69ef42d09bfad49bbdddaa4a [CANDIDATE]: Term 1 election: Requested vote from peers 218aa16cad18435d8f9dec9ba96839ca (127.0.98.2:38101), f41e17f9fc744806a4333399545da3e1 (127.0.98.3:38303)
I20260929 19:04:32.301077 17685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a81592216cd047d6bf03992a97afac0d" candidate_uuid: "4641ddfd69ef42d09bfad49bbdddaa4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "218aa16cad18435d8f9dec9ba96839ca"
I20260929 19:04:32.301102 17819 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a81592216cd047d6bf03992a97afac0d" candidate_uuid: "4641ddfd69ef42d09bfad49bbdddaa4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f41e17f9fc744806a4333399545da3e1"
I20260929 19:04:32.301146 17685 raft_consensus.cc:3037] T a81592216cd047d6bf03992a97afac0d P 218aa16cad18435d8f9dec9ba96839ca [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:04:32.301179 17819 raft_consensus.cc:3037] T a81592216cd047d6bf03992a97afac0d P f41e17f9fc744806a4333399545da3e1 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:04:32.301800 17685 raft_consensus.cc:2454] T a81592216cd047d6bf03992a97afac0d P 218aa16cad18435d8f9dec9ba96839ca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4641ddfd69ef42d09bfad49bbdddaa4a in term 1.
I20260929 19:04:32.301887 17819 raft_consensus.cc:2454] T a81592216cd047d6bf03992a97afac0d P f41e17f9fc744806a4333399545da3e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4641ddfd69ef42d09bfad49bbdddaa4a in term 1.
I20260929 19:04:32.302022 17487 leader_election.cc:302] T a81592216cd047d6bf03992a97afac0d P 4641ddfd69ef42d09bfad49bbdddaa4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 218aa16cad18435d8f9dec9ba96839ca, 4641ddfd69ef42d09bfad49bbdddaa4a; no voters:
I20260929 19:04:32.302134 17885 raft_consensus.cc:2781] T a81592216cd047d6bf03992a97afac0d P 4641ddfd69ef42d09bfad49bbdddaa4a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:04:32.302286 17885 raft_consensus.cc:686] T a81592216cd047d6bf03992a97afac0d P 4641ddfd69ef42d09bfad49bbdddaa4a [term 1 LEADER]: Becoming Leader. State: Replica: 4641ddfd69ef42d09bfad49bbdddaa4a, State: Running, Role: LEADER
I20260929 19:04:32.302379 17885 consensus_queue.cc:237] T a81592216cd047d6bf03992a97afac0d P 4641ddfd69ef42d09bfad49bbdddaa4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4641ddfd69ef42d09bfad49bbdddaa4a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45117 } } peers { permanent_uuid: "218aa16cad18435d8f9dec9ba96839ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38101 } } peers { permanent_uuid: "f41e17f9fc744806a4333399545da3e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38303 } }
I20260929 19:04:32.303143 17406 catalog_manager.cc:5495] T a81592216cd047d6bf03992a97afac0d P 4641ddfd69ef42d09bfad49bbdddaa4a reported cstate change: term changed from 0 to 1, leader changed from <none> to 4641ddfd69ef42d09bfad49bbdddaa4a (127.0.98.1). New cstate: current_term: 1 leader_uuid: "4641ddfd69ef42d09bfad49bbdddaa4a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4641ddfd69ef42d09bfad49bbdddaa4a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "218aa16cad18435d8f9dec9ba96839ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f41e17f9fc744806a4333399545da3e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38303 } health_report { overall_health: UNKNOWN } } }
I20260929 19:04:32.308099 17875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:32.305789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39670 (local address 127.0.98.1:45117)
0929 19:04:32.305825 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0929 19:04:32.305827 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:04:32.305835 (+ 8us) server_negotiation.cc:407] Connection header received
0929 19:04:32.305903 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:04:32.305903 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:04:32.305919 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:04:32.306036 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 19:04:32.306155 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:32.306589 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:32.307343 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:32.307538 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:32.307609 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:04:32.307775 (+ 166us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 19:04:32.307874 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 19:04:32.307952 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0929 19:04:32.307977 (+ 25us) server_negotiation.cc:299] Negotiation successful
0929 19:04:32.307995 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260929 19:04:32.309562 17685 raft_consensus.cc:1261] T a81592216cd047d6bf03992a97afac0d P 218aa16cad18435d8f9dec9ba96839ca [term 1 FOLLOWER]: Refusing update from remote peer 4641ddfd69ef42d09bfad49bbdddaa4a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 19:04:32.309653 17819 raft_consensus.cc:1261] T a81592216cd047d6bf03992a97afac0d P f41e17f9fc744806a4333399545da3e1 [term 1 FOLLOWER]: Refusing update from remote peer 4641ddfd69ef42d09bfad49bbdddaa4a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 19:04:32.309872 17885 consensus_queue.cc:1035] T a81592216cd047d6bf03992a97afac0d P 4641ddfd69ef42d09bfad49bbdddaa4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f41e17f9fc744806a4333399545da3e1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 19:04:32.309993 17889 consensus_queue.cc:1035] T a81592216cd047d6bf03992a97afac0d P 4641ddfd69ef42d09bfad49bbdddaa4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "218aa16cad18435d8f9dec9ba96839ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 19:04:32.312153 17893 mvcc.cc:205] Tried to move back new op lower bound from 7334742721777762304 to 7334742721750966272. Current Snapshot: MvccSnapshot[applied={T|T < 7334742721777762304}]
I20260929 19:04:32.312538 17894 mvcc.cc:205] Tried to move back new op lower bound from 7334742721777762304 to 7334742721750966272. Current Snapshot: MvccSnapshot[applied={T|T < 7334742721777762304}]
I20260929 19:04:32.313887 17405 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60578:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 19:04:32.313968 17405 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60578:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 19:04:32.314594 17405 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5eae6349988c4802981353bc907e10d5: Sending DeleteTablet for 3 replicas of tablet a81592216cd047d6bf03992a97afac0d
I20260929 19:04:32.314841 17531 tablet_service.cc:1514] Processing DeleteTablet for tablet a81592216cd047d6bf03992a97afac0d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:04:32 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39664
I20260929 19:04:32.314895 17665 tablet_service.cc:1514] Processing DeleteTablet for tablet a81592216cd047d6bf03992a97afac0d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:04:32 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39198
I20260929 19:04:32.315042 17902 tablet_replica.cc:330] T a81592216cd047d6bf03992a97afac0d P 4641ddfd69ef42d09bfad49bbdddaa4a: stopping tablet replica
I20260929 19:04:32.315050 17799 tablet_service.cc:1514] Processing DeleteTablet for tablet a81592216cd047d6bf03992a97afac0d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:04:32 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38606
I20260929 19:04:32.315184 17902 raft_consensus.cc:2229] T a81592216cd047d6bf03992a97afac0d P 4641ddfd69ef42d09bfad49bbdddaa4a [term 1 LEADER]: Raft consensus shutting down.
I20260929 19:04:32.315301 17903 tablet_replica.cc:330] T a81592216cd047d6bf03992a97afac0d P 218aa16cad18435d8f9dec9ba96839ca: stopping tablet replica
I20260929 19:04:32.315299 17902 raft_consensus.cc:2258] T a81592216cd047d6bf03992a97afac0d P 4641ddfd69ef42d09bfad49bbdddaa4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 19:04:32.315454 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUbnLHE/build/release/bin/kudu with pid 17468
I20260929 19:04:32.317694 17903 raft_consensus.cc:2229] T a81592216cd047d6bf03992a97afac0d P 218aa16cad18435d8f9dec9ba96839ca [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 19:04:32.317811 17903 raft_consensus.cc:2258] T a81592216cd047d6bf03992a97afac0d P 218aa16cad18435d8f9dec9ba96839ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 19:04:32.318243 17903 ts_tablet_manager.cc:1905] T a81592216cd047d6bf03992a97afac0d P 218aa16cad18435d8f9dec9ba96839ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 19:04:32.320055 17903 ts_tablet_manager.cc:1918] T a81592216cd047d6bf03992a97afac0d P 218aa16cad18435d8f9dec9ba96839ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 19:04:32.320133 17903 log.cc:1196] T a81592216cd047d6bf03992a97afac0d P 218aa16cad18435d8f9dec9ba96839ca: Deleting WAL directory at /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708447806542-392-0/minicluster-data/ts-1/wal/wals/a81592216cd047d6bf03992a97afac0d
I20260929 19:04:32.315237 17904 tablet_replica.cc:330] T a81592216cd047d6bf03992a97afac0d P f41e17f9fc744806a4333399545da3e1: stopping tablet replica
I20260929 19:04:32.320315 17904 raft_consensus.cc:2229] T a81592216cd047d6bf03992a97afac0d P f41e17f9fc744806a4333399545da3e1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 19:04:32.320420 17903 ts_tablet_manager.cc:1939] T a81592216cd047d6bf03992a97afac0d P 218aa16cad18435d8f9dec9ba96839ca: Deleting consensus metadata
I20260929 19:04:32.320432 17904 raft_consensus.cc:2258] T a81592216cd047d6bf03992a97afac0d P f41e17f9fc744806a4333399545da3e1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 19:04:32.320876 17904 ts_tablet_manager.cc:1905] T a81592216cd047d6bf03992a97afac0d P f41e17f9fc744806a4333399545da3e1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 19:04:32.320885 17393 catalog_manager.cc:4841] TS 218aa16cad18435d8f9dec9ba96839ca (127.0.98.2:38101): tablet a81592216cd047d6bf03992a97afac0d (table test-table [id=b6f086f9e7d64c5c9ffb0416786351ca]) successfully deleted
W20260929 19:04:32.321338 17393 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:45117 (error 108)
I20260929 19:04:32.321542 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUbnLHE/build/release/bin/kudu with pid 17602
I20260929 19:04:32.322108 17876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:32.321627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45117 (local address 127.0.0.1:39686)
0929 19:04:32.322042 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0929 19:04:32.322075 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45117: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":399}
I20260929 19:04:32.322515 17904 ts_tablet_manager.cc:1918] T a81592216cd047d6bf03992a97afac0d P f41e17f9fc744806a4333399545da3e1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 19:04:32.322556 17904 log.cc:1196] T a81592216cd047d6bf03992a97afac0d P f41e17f9fc744806a4333399545da3e1: Deleting WAL directory at /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790708447806542-392-0/minicluster-data/ts-2/wal/wals/a81592216cd047d6bf03992a97afac0d
I20260929 19:04:32.322719 17904 ts_tablet_manager.cc:1939] T a81592216cd047d6bf03992a97afac0d P f41e17f9fc744806a4333399545da3e1: Deleting consensus metadata
I20260929 19:04:32.323268 17392 catalog_manager.cc:4841] TS f41e17f9fc744806a4333399545da3e1 (127.0.98.3:38303): tablet a81592216cd047d6bf03992a97afac0d (table test-table [id=b6f086f9e7d64c5c9ffb0416786351ca]) successfully deleted
W20260929 19:04:32.323655 17393 catalog_manager.cc:4584] TS 4641ddfd69ef42d09bfad49bbdddaa4a (127.0.98.1:45117): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a81592216cd047d6bf03992a97afac0d: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45117: connect: Connection refused (error 111)
I20260929 19:04:32.326681 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUbnLHE/build/release/bin/kudu with pid 17736
I20260929 19:04:32.330837 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUbnLHE/build/release/bin/kudu with pid 17374
2026-09-29T19:04:32Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (1795 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-29T19:04:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T19:04:32Z Disabled control of system clock
I20260929 19:04:32.356177 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUbnLHE/build/release/bin/kudu
/tmp/dist-test-taskUbnLHE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708447806542-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708447806542-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708447806542-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708447806542-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33229
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:40979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33229
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 19:04:32.430895 17909 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:04:32.431082 17909 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:04:32.431100 17909 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:04:32.432156 17909 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 19:04:32.432193 17909 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:04:32.432204 17909 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:04:32.432238 17909 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 19:04:32.432263 17909 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 19:04:32.433446 17909 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708447806542-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708447806542-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33229
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33229
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708447806542-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17909
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708447806542-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 18:09:13 UTC on ccycloud.quasar-hpkalf.root.comops.site
build id 4680913
I20260929 19:04:32.433697 17909 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:04:32.433951 17909 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:04:32.438881 17915 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:04:32.438920 17909 server_base.cc:1034] running on GCE node
W20260929 19:04:32.438885 17917 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:04:32.438935 17914 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:04:32.439278 17909 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:04:32.439574 17909 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:04:32.440730 17909 hybrid_clock.cc:648] HybridClock initialized: now 1790708672440724 us; error 33 us; skew 500 ppm
I20260929 19:04:32.441885 17909 webserver.cc:460] Webserver started at http://127.0.98.62:35383/ using document root <none> and password file <none>
I20260929 19:04:32.442097 17909 fs_manager.cc:359] Metadata directory not provided
I20260929 19:04:32.442176 17909 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:04:32.442329 17909 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:04:32.443262 17909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708447806542-392-0/minicluster-data/master-0/data/instance:
uuid: "254e493dfb484453b093430efb0bd798"
format_stamp: "Formatted at 2026-09-29 19:04:32 on dist-test-slave-lqkv"
server_key: "7a9e0f67e76091c30bc9f8774fd12511"
server_key_iv: "aeef94a76cc014f5725e6075122ae6a9"
server_key_version: "encryptionkey@0"
I20260929 19:04:32.443591 17909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708447806542-392-0/minicluster-data/master-0/wal/instance:
uuid: "254e493dfb484453b093430efb0bd798"
format_stamp: "Formatted at 2026-09-29 19:04:32 on dist-test-slave-lqkv"
server_key: "7a9e0f67e76091c30bc9f8774fd12511"
server_key_iv: "aeef94a76cc014f5725e6075122ae6a9"
server_key_version: "encryptionkey@0"
I20260929 19:04:32.444849 17909 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 19:04:32.445652 17923 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:04:32.445822 17909 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 19:04:32.445915 17909 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708447806542-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708447806542-392-0/minicluster-data/master-0/wal
uuid: "254e493dfb484453b093430efb0bd798"
format_stamp: "Formatted at 2026-09-29 19:04:32 on dist-test-slave-lqkv"
server_key: "7a9e0f67e76091c30bc9f8774fd12511"
server_key_iv: "aeef94a76cc014f5725e6075122ae6a9"
server_key_version: "encryptionkey@0"
I20260929 19:04:32.446015 17909 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708447806542-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708447806542-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708447806542-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:04:32.464605 17909 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:04:32.464854 17909 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:04:32.465013 17909 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:04:32.468523 17909 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33229
I20260929 19:04:32.468575 17975 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33229 every 8 connection(s)
I20260929 19:04:32.468885 17909 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708447806542-392-0/minicluster-data/master-0/data/info.pb
I20260929 19:04:32.469733 17976 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:04:32.469838 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUbnLHE/build/release/bin/kudu as pid 17909
I20260929 19:04:32.469981 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708447806542-392-0/minicluster-data/master-0/wal/instance
I20260929 19:04:32.470175 392 external_mini_cluster.cc:1428] Setting key 50b4254dcd4abbe921e3d25d65fb0f3b
I20260929 19:04:32.472975 17976 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 254e493dfb484453b093430efb0bd798: Bootstrap starting.
I20260929 19:04:32.473515 17976 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 254e493dfb484453b093430efb0bd798: Neither blocks nor log segments found. Creating new log.
I20260929 19:04:32.473786 17976 log.cc:824] T 00000000000000000000000000000000 P 254e493dfb484453b093430efb0bd798: Log is configured to *not* fsync() on all Append() calls
I20260929 19:04:32.474465 17976 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 254e493dfb484453b093430efb0bd798: No bootstrap required, opened a new log
I20260929 19:04:32.475740 17979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:32.470647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33762 (local address 127.0.98.62:33229)
0929 19:04:32.470866 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0929 19:04:32.470873 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 19:04:32.470904 (+ 31us) server_negotiation.cc:407] Connection header received
0929 19:04:32.471394 (+ 490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:04:32.471402 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:04:32.471475 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:04:32.471615 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:04:32.471801 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:32.473080 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:32.473832 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:32.474355 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:32.474478 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:04:32.475120 (+ 642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:04:32.475123 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:04:32.475124 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 19:04:32.475304 (+ 180us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:04:32.475350 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 19:04:32.475362 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:04:32.475430 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0929 19:04:32.475487 (+ 57us) server_negotiation.cc:299] Negotiation successful
0929 19:04:32.475557 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":84,"threads_started":1}
I20260929 19:04:32.476615 17976 raft_consensus.cc:348] T 00000000000000000000000000000000 P 254e493dfb484453b093430efb0bd798 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "254e493dfb484453b093430efb0bd798" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33229 } }
I20260929 19:04:32.476730 17976 raft_consensus.cc:374] T 00000000000000000000000000000000 P 254e493dfb484453b093430efb0bd798 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:04:32.476751 17976 raft_consensus.cc:729] T 00000000000000000000000000000000 P 254e493dfb484453b093430efb0bd798 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 254e493dfb484453b093430efb0bd798, State: Initialized, Role: FOLLOWER
I20260929 19:04:32.476919 17976 consensus_queue.cc:260] T 00000000000000000000000000000000 P 254e493dfb484453b093430efb0bd798 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "254e493dfb484453b093430efb0bd798" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33229 } }
I20260929 19:04:32.476981 17976 raft_consensus.cc:388] T 00000000000000000000000000000000 P 254e493dfb484453b093430efb0bd798 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 19:04:32.477023 17976 raft_consensus.cc:482] T 00000000000000000000000000000000 P 254e493dfb484453b093430efb0bd798 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 19:04:32.477069 17976 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 254e493dfb484453b093430efb0bd798 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:04:32.477547 17976 raft_consensus.cc:504] T 00000000000000000000000000000000 P 254e493dfb484453b093430efb0bd798 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "254e493dfb484453b093430efb0bd798" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33229 } }
I20260929 19:04:32.477651 17976 leader_election.cc:302] T 00000000000000000000000000000000 P 254e493dfb484453b093430efb0bd798 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 254e493dfb484453b093430efb0bd798; no voters:
I20260929 19:04:32.477778 17976 leader_election.cc:288] T 00000000000000000000000000000000 P 254e493dfb484453b093430efb0bd798 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 19:04:32.477825 17981 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 254e493dfb484453b093430efb0bd798 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:04:32.477983 17981 raft_consensus.cc:686] T 00000000000000000000000000000000 P 254e493dfb484453b093430efb0bd798 [term 1 LEADER]: Becoming Leader. State: Replica: 254e493dfb484453b093430efb0bd798, State: Running, Role: LEADER
I20260929 19:04:32.478021 17976 sys_catalog.cc:564] T 00000000000000000000000000000000 P 254e493dfb484453b093430efb0bd798 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 19:04:32.478109 17981 consensus_queue.cc:237] T 00000000000000000000000000000000 P 254e493dfb484453b093430efb0bd798 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "254e493dfb484453b093430efb0bd798" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33229 } }
I20260929 19:04:32.478421 17982 sys_catalog.cc:455] T 00000000000000000000000000000000 P 254e493dfb484453b093430efb0bd798 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "254e493dfb484453b093430efb0bd798" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "254e493dfb484453b093430efb0bd798" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33229 } } }
I20260929 19:04:32.478510 17982 sys_catalog.cc:458] T 00000000000000000000000000000000 P 254e493dfb484453b093430efb0bd798 [sys.catalog]: This master's current role is: LEADER
I20260929 19:04:32.478427 17983 sys_catalog.cc:455] T 00000000000000000000000000000000 P 254e493dfb484453b093430efb0bd798 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 254e493dfb484453b093430efb0bd798. Latest consensus state: current_term: 1 leader_uuid: "254e493dfb484453b093430efb0bd798" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "254e493dfb484453b093430efb0bd798" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33229 } } }
I20260929 19:04:32.478631 17983 sys_catalog.cc:458] T 00000000000000000000000000000000 P 254e493dfb484453b093430efb0bd798 [sys.catalog]: This master's current role is: LEADER
I20260929 19:04:32.478708 17989 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 19:04:32.479167 17989 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 19:04:32.480332 17989 catalog_manager.cc:1347] Generated new cluster ID: 5dd8d798a22245b684af70087f1ccbaa
I20260929 19:04:32.480386 17989 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 19:04:32.484027 17989 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 19:04:32.484443 17989 catalog_manager.cc:1504] Loading token signing keys...
I20260929 19:04:32.487092 17989 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 254e493dfb484453b093430efb0bd798: Generated new TSK 0
I20260929 19:04:32.487265 17989 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 19:04:32.490680 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUbnLHE/build/release/bin/kudu
/tmp/dist-test-taskUbnLHE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708447806542-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708447806542-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708447806542-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708447806542-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33229
--builtin_ntp_servers=127.0.98.20:40979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 19:04:32.571213 18000 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:04:32.571409 18000 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:04:32.571429 18000 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:04:32.572431 18000 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:04:32.572461 18000 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:04:32.572484 18000 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 19:04:32.573700 18000 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708447806542-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708447806542-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708447806542-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33229
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18000
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708447806542-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 18:09:13 UTC on ccycloud.quasar-hpkalf.root.comops.site
build id 4680913
I20260929 19:04:32.573855 18000 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:04:32.574059 18000 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:04:32.579690 18008 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:04:32.579702 18006 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:04:32.579873 18000 server_base.cc:1034] running on GCE node
W20260929 19:04:32.580030 18005 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:04:32.580248 18000 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:04:32.580507 18000 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:04:32.581650 18000 hybrid_clock.cc:648] HybridClock initialized: now 1790708672581639 us; error 25 us; skew 500 ppm
I20260929 19:04:32.582897 18000 webserver.cc:460] Webserver started at http://127.0.98.1:36155/ using document root <none> and password file <none>
I20260929 19:04:32.583099 18000 fs_manager.cc:359] Metadata directory not provided
I20260929 19:04:32.583139 18000 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:04:32.583220 18000 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:04:32.584127 18000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708447806542-392-0/minicluster-data/ts-0/data/instance:
uuid: "8cfb18cc973d4be39b49273acb1f40b5"
format_stamp: "Formatted at 2026-09-29 19:04:32 on dist-test-slave-lqkv"
server_key: "6ada6a26c62723f0412ffff7b42edf37"
server_key_iv: "2ef29d8948c288aad7c745fd70ffc0ac"
server_key_version: "encryptionkey@0"
I20260929 19:04:32.584406 18000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708447806542-392-0/minicluster-data/ts-0/wal/instance:
uuid: "8cfb18cc973d4be39b49273acb1f40b5"
format_stamp: "Formatted at 2026-09-29 19:04:32 on dist-test-slave-lqkv"
server_key: "6ada6a26c62723f0412ffff7b42edf37"
server_key_iv: "2ef29d8948c288aad7c745fd70ffc0ac"
server_key_version: "encryptionkey@0"
I20260929 19:04:32.585625 18000 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 19:04:32.586302 18014 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:04:32.586452 18000 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 19:04:32.586529 18000 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708447806542-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708447806542-392-0/minicluster-data/ts-0/wal
uuid: "8cfb18cc973d4be39b49273acb1f40b5"
format_stamp: "Formatted at 2026-09-29 19:04:32 on dist-test-slave-lqkv"
server_key: "6ada6a26c62723f0412ffff7b42edf37"
server_key_iv: "2ef29d8948c288aad7c745fd70ffc0ac"
server_key_version: "encryptionkey@0"
I20260929 19:04:32.586576 18000 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708447806542-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708447806542-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708447806542-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:04:32.595631 18000 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:04:32.595867 18000 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:04:32.596035 18000 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:04:32.596263 18000 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:04:32.596540 18000 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:04:32.596592 18000 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:04:32.596630 18000 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:04:32.596657 18000 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:04:32.603003 18000 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39269
I20260929 19:04:32.603041 18127 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39269 every 8 connection(s)
I20260929 19:04:32.603327 18000 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708447806542-392-0/minicluster-data/ts-0/data/info.pb
I20260929 19:04:32.605242 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUbnLHE/build/release/bin/kudu as pid 18000
I20260929 19:04:32.605345 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708447806542-392-0/minicluster-data/ts-0/wal/instance
I20260929 19:04:32.605532 392 external_mini_cluster.cc:1428] Setting key 40f0400cec0d09da6b05d5dd9e04f51d
I20260929 19:04:32.606444 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUbnLHE/build/release/bin/kudu
/tmp/dist-test-taskUbnLHE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708447806542-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708447806542-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708447806542-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708447806542-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33229
--builtin_ntp_servers=127.0.98.20:40979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 19:04:32.608664 17979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:32.603933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56065 (local address 127.0.98.62:33229)
0929 19:04:32.603969 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0929 19:04:32.603971 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:04:32.604138 (+ 167us) server_negotiation.cc:407] Connection header received
0929 19:04:32.604575 (+ 437us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:04:32.604576 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:04:32.604590 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:04:32.604634 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:04:32.605402 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:32.606112 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:32.607161 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:32.607506 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:32.607583 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:04:32.608424 (+ 841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:04:32.608425 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:04:32.608425 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 19:04:32.608429 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:04:32.608439 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 19:04:32.608443 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:04:32.608495 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0929 19:04:32.608593 (+ 98us) server_negotiation.cc:299] Negotiation successful
0929 19:04:32.608614 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260929 19:04:32.608991 18130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:32.603899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33229 (local address 127.0.98.1:56065)
0929 19:04:32.604041 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0929 19:04:32.604071 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0929 19:04:32.604479 (+ 408us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:04:32.604686 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:04:32.604689 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:04:32.604697 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:04:32.605274 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:04:32.605280 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:04:32.606206 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:04:32.606207 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:04:32.607079 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:04:32.607082 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:04:32.607605 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:04:32.607606 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:04:32.607770 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:04:32.608259 (+ 489us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 19:04:32.608260 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:04:32.608296 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 19:04:32.608324 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:04:32.608511 (+ 187us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:04:32.608515 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:04:32.608517 (+ 2us) client_negotiation.cc:769] Sending connection context
0929 19:04:32.608590 (+ 73us) client_negotiation.cc:240] Negotiation successful
0929 19:04:32.608682 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87,"thread_start_us":60,"threads_started":1}
I20260929 19:04:32.609481 18128 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33229
I20260929 19:04:32.609572 18128 heartbeater.cc:464] Registering TS with master...
I20260929 19:04:32.609736 18128 heartbeater.cc:511] Master 127.0.98.62:33229 requested a full tablet report, sending...
I20260929 19:04:32.610235 17940 ts_manager.cc:194] Registered new tserver with Master: 8cfb18cc973d4be39b49273acb1f40b5 (127.0.98.1:39269)
I20260929 19:04:32.611168 17940 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:56065
W20260929 19:04:32.682526 18131 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:04:32.682693 18131 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:04:32.682710 18131 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:04:32.683688 18131 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:04:32.683718 18131 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:04:32.683753 18131 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 19:04:32.684890 18131 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708447806542-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708447806542-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708447806542-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33229
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18131
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708447806542-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 18:09:13 UTC on ccycloud.quasar-hpkalf.root.comops.site
build id 4680913
I20260929 19:04:32.685041 18131 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:04:32.685236 18131 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:04:32.689724 18139 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:04:32.689817 18131 server_base.cc:1034] running on GCE node
W20260929 19:04:32.689821 18137 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:04:32.689718 18136 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:04:32.690254 18131 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:04:32.690477 18131 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:04:32.691615 18131 hybrid_clock.cc:648] HybridClock initialized: now 1790708672691600 us; error 31 us; skew 500 ppm
I20260929 19:04:32.692672 18131 webserver.cc:460] Webserver started at http://127.0.98.2:46417/ using document root <none> and password file <none>
I20260929 19:04:32.692896 18131 fs_manager.cc:359] Metadata directory not provided
I20260929 19:04:32.692938 18131 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:04:32.693017 18131 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:04:32.693924 18131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708447806542-392-0/minicluster-data/ts-1/data/instance:
uuid: "2c0a4d9e5e8c4b95b2d21e953a528e5f"
format_stamp: "Formatted at 2026-09-29 19:04:32 on dist-test-slave-lqkv"
server_key: "92169bbb22299b97ff9576927ad67e8a"
server_key_iv: "c3bad364def1e902dfc2dff42caf988b"
server_key_version: "encryptionkey@0"
I20260929 19:04:32.694185 18131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708447806542-392-0/minicluster-data/ts-1/wal/instance:
uuid: "2c0a4d9e5e8c4b95b2d21e953a528e5f"
format_stamp: "Formatted at 2026-09-29 19:04:32 on dist-test-slave-lqkv"
server_key: "92169bbb22299b97ff9576927ad67e8a"
server_key_iv: "c3bad364def1e902dfc2dff42caf988b"
server_key_version: "encryptionkey@0"
I20260929 19:04:32.695271 18131 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 19:04:32.695825 18145 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:04:32.695990 18131 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 19:04:32.696094 18131 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708447806542-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708447806542-392-0/minicluster-data/ts-1/wal
uuid: "2c0a4d9e5e8c4b95b2d21e953a528e5f"
format_stamp: "Formatted at 2026-09-29 19:04:32 on dist-test-slave-lqkv"
server_key: "92169bbb22299b97ff9576927ad67e8a"
server_key_iv: "c3bad364def1e902dfc2dff42caf988b"
server_key_version: "encryptionkey@0"
I20260929 19:04:32.696171 18131 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708447806542-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708447806542-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708447806542-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:04:32.711513 18131 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:04:32.711747 18131 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:04:32.711875 18131 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:04:32.712080 18131 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:04:32.712353 18131 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:04:32.712404 18131 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:04:32.712438 18131 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:04:32.712464 18131 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:04:32.717938 18131 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42487
I20260929 19:04:32.718005 18258 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42487 every 8 connection(s)
I20260929 19:04:32.718302 18131 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708447806542-392-0/minicluster-data/ts-1/data/info.pb
I20260929 19:04:32.721194 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUbnLHE/build/release/bin/kudu as pid 18131
I20260929 19:04:32.721308 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708447806542-392-0/minicluster-data/ts-1/wal/instance
I20260929 19:04:32.721508 392 external_mini_cluster.cc:1428] Setting key b83cb1910803b1bdd5bf5cb850fc54a0
I20260929 19:04:32.722412 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUbnLHE/build/release/bin/kudu
/tmp/dist-test-taskUbnLHE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708447806542-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708447806542-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708447806542-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708447806542-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33229
--builtin_ntp_servers=127.0.98.20:40979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 19:04:32.723402 17979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:32.718956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41975 (local address 127.0.98.62:33229)
0929 19:04:32.719036 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0929 19:04:32.719039 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 19:04:32.719208 (+ 169us) server_negotiation.cc:407] Connection header received
0929 19:04:32.719618 (+ 410us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:04:32.719620 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:04:32.719628 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:04:32.719674 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:04:32.720202 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:32.720672 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:32.721988 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:32.722269 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:32.722351 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:04:32.723173 (+ 822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:04:32.723174 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:04:32.723174 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 19:04:32.723178 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:04:32.723191 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 19:04:32.723195 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:04:32.723241 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0929 19:04:32.723303 (+ 62us) server_negotiation.cc:299] Negotiation successful
0929 19:04:32.723327 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 19:04:32.723661 18261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:32.718926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33229 (local address 127.0.98.2:41975)
0929 19:04:32.719100 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0929 19:04:32.719130 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0929 19:04:32.719549 (+ 419us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:04:32.719717 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:04:32.719719 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:04:32.719727 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:04:32.720137 (+ 410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:04:32.720141 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:04:32.720741 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:04:32.720741 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:04:32.721896 (+ 1155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:04:32.721899 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:04:32.722382 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:04:32.722383 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:04:32.722509 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:04:32.723021 (+ 512us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 19:04:32.723021 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:04:32.723046 (+ 25us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 19:04:32.723065 (+ 19us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:04:32.723241 (+ 176us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:04:32.723245 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:04:32.723248 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 19:04:32.723308 (+ 60us) client_negotiation.cc:240] Negotiation successful
0929 19:04:32.723402 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":97,"threads_started":1}
I20260929 19:04:32.724185 18259 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33229
I20260929 19:04:32.724280 18259 heartbeater.cc:464] Registering TS with master...
I20260929 19:04:32.724442 18259 heartbeater.cc:511] Master 127.0.98.62:33229 requested a full tablet report, sending...
I20260929 19:04:32.724783 17940 ts_manager.cc:194] Registered new tserver with Master: 2c0a4d9e5e8c4b95b2d21e953a528e5f (127.0.98.2:42487)
I20260929 19:04:32.725180 17940 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:41975
W20260929 19:04:32.796866 18262 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:04:32.797065 18262 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:04:32.797101 18262 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:04:32.798187 18262 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:04:32.798242 18262 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:04:32.798280 18262 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 19:04:32.799440 18262 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708447806542-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708447806542-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708447806542-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33229
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18262
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708447806542-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 18:09:13 UTC on ccycloud.quasar-hpkalf.root.comops.site
build id 4680913
I20260929 19:04:32.799659 18262 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:04:32.799901 18262 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:04:32.804455 18268 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:04:32.804495 18262 server_base.cc:1034] running on GCE node
W20260929 19:04:32.804477 18270 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:04:32.804461 18267 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:04:32.804898 18262 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:04:32.805136 18262 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:04:32.806303 18262 hybrid_clock.cc:648] HybridClock initialized: now 1790708672806288 us; error 40 us; skew 500 ppm
I20260929 19:04:32.807515 18262 webserver.cc:460] Webserver started at http://127.0.98.3:43841/ using document root <none> and password file <none>
I20260929 19:04:32.807742 18262 fs_manager.cc:359] Metadata directory not provided
I20260929 19:04:32.807811 18262 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:04:32.807937 18262 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:04:32.809002 18262 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708447806542-392-0/minicluster-data/ts-2/data/instance:
uuid: "1ff6e66969c94788954ea1bceb79cc18"
format_stamp: "Formatted at 2026-09-29 19:04:32 on dist-test-slave-lqkv"
server_key: "a5ff433cdabc2895029598e9d944a836"
server_key_iv: "a10bbb81e41048ae91a97ad72a7f3f48"
server_key_version: "encryptionkey@0"
I20260929 19:04:32.809306 18262 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708447806542-392-0/minicluster-data/ts-2/wal/instance:
uuid: "1ff6e66969c94788954ea1bceb79cc18"
format_stamp: "Formatted at 2026-09-29 19:04:32 on dist-test-slave-lqkv"
server_key: "a5ff433cdabc2895029598e9d944a836"
server_key_iv: "a10bbb81e41048ae91a97ad72a7f3f48"
server_key_version: "encryptionkey@0"
I20260929 19:04:32.810491 18262 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260929 19:04:32.811160 18276 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:04:32.811357 18262 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 19:04:32.811434 18262 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708447806542-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708447806542-392-0/minicluster-data/ts-2/wal
uuid: "1ff6e66969c94788954ea1bceb79cc18"
format_stamp: "Formatted at 2026-09-29 19:04:32 on dist-test-slave-lqkv"
server_key: "a5ff433cdabc2895029598e9d944a836"
server_key_iv: "a10bbb81e41048ae91a97ad72a7f3f48"
server_key_version: "encryptionkey@0"
I20260929 19:04:32.811496 18262 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708447806542-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708447806542-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708447806542-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:04:32.827191 18262 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:04:32.827466 18262 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:04:32.827598 18262 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:04:32.827791 18262 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:04:32.828070 18262 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:04:32.828123 18262 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:04:32.828161 18262 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:04:32.828186 18262 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:04:32.833325 18262 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37259
I20260929 19:04:32.833359 18389 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37259 every 8 connection(s)
I20260929 19:04:32.833640 18262 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708447806542-392-0/minicluster-data/ts-2/data/info.pb
I20260929 19:04:32.837188 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUbnLHE/build/release/bin/kudu as pid 18262
I20260929 19:04:32.837301 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708447806542-392-0/minicluster-data/ts-2/wal/instance
I20260929 19:04:32.837478 392 external_mini_cluster.cc:1428] Setting key 8fd56916f09602bf28bfb2c3f36e821c
I20260929 19:04:32.838604 17979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:32.834285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:32847 (local address 127.0.98.62:33229)
0929 19:04:32.834353 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0929 19:04:32.834355 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:04:32.834549 (+ 194us) server_negotiation.cc:407] Connection header received
0929 19:04:32.834969 (+ 420us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:04:32.834970 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:04:32.834980 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:04:32.835027 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:04:32.835680 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:32.836163 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:32.837236 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:32.837480 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:32.837554 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:04:32.838382 (+ 828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:04:32.838383 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:04:32.838384 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 19:04:32.838388 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:04:32.838400 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 19:04:32.838404 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:04:32.838445 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0929 19:04:32.838514 (+ 69us) server_negotiation.cc:299] Negotiation successful
0929 19:04:32.838534 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260929 19:04:32.838791 18392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:32.834275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33229 (local address 127.0.98.3:32847)
0929 19:04:32.834464 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0929 19:04:32.834491 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0929 19:04:32.834893 (+ 402us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:04:32.835072 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:04:32.835074 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:04:32.835082 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:04:32.835613 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:04:32.835617 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:04:32.836235 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:04:32.836236 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:04:32.837152 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:04:32.837155 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:04:32.837573 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:04:32.837574 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:04:32.837723 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:04:32.838242 (+ 519us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 19:04:32.838243 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:04:32.838277 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 19:04:32.838301 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:04:32.838445 (+ 144us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:04:32.838450 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:04:32.838452 (+ 2us) client_negotiation.cc:769] Sending connection context
0929 19:04:32.838514 (+ 62us) client_negotiation.cc:240] Negotiation successful
0929 19:04:32.838598 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":96,"threads_started":1}
I20260929 19:04:32.839202 18390 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33229
I20260929 19:04:32.839265 18390 heartbeater.cc:464] Registering TS with master...
I20260929 19:04:32.839387 18390 heartbeater.cc:511] Master 127.0.98.62:33229 requested a full tablet report, sending...
I20260929 19:04:32.839659 17940 ts_manager.cc:194] Registered new tserver with Master: 1ff6e66969c94788954ea1bceb79cc18 (127.0.98.3:37259)
I20260929 19:04:32.839999 17940 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:32847
I20260929 19:04:32.849059 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 19:04:32.852926 17979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:32.850239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33770 (local address 127.0.98.62:33229)
0929 19:04:32.850307 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0929 19:04:32.850308 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 19:04:32.850401 (+ 93us) server_negotiation.cc:407] Connection header received
0929 19:04:32.850460 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:04:32.850461 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:04:32.850470 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:04:32.850522 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:04:32.850703 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:32.851131 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:32.851843 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:32.852022 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:32.852085 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:04:32.852693 (+ 608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:04:32.852694 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:04:32.852694 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 19:04:32.852698 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:04:32.852708 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 19:04:32.852711 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:04:32.852752 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0929 19:04:32.852833 (+ 81us) server_negotiation.cc:299] Negotiation successful
0929 19:04:32.852856 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 19:04:32.854234 17940 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33770:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 19:04:32.855722 17940 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 19:04:32.861687 18401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:32.857619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39269 (local address 127.0.0.1:58204)
0929 19:04:32.858069 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0929 19:04:32.858080 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 19:04:32.858136 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:04:32.858308 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:04:32.858309 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:04:32.858311 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:04:32.858355 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:04:32.858356 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:04:32.859618 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:04:32.859619 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:04:32.860488 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:04:32.860491 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:04:32.861406 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:04:32.861407 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:04:32.861480 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:04:32.861490 (+ 10us) client_negotiation.cc:769] Sending connection context
0929 19:04:32.861567 (+ 77us) client_negotiation.cc:240] Negotiation successful
0929 19:04:32.861617 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":77,"threads_started":1}
I20260929 19:04:32.861707 18402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:32.857900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58204 (local address 127.0.98.1:39269)
0929 19:04:32.858065 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0929 19:04:32.858069 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 19:04:32.858130 (+ 61us) server_negotiation.cc:407] Connection header received
0929 19:04:32.858199 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:04:32.858201 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:04:32.858217 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:04:32.858294 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:04:32.858515 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:32.859540 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:32.860548 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:32.861298 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:32.861386 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:04:32.861475 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0929 19:04:32.861582 (+ 107us) server_negotiation.cc:299] Negotiation successful
0929 19:04:32.861614 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":78,"threads_started":1}
I20260929 19:04:32.861857 18403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:32.857934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47648 (local address 127.0.98.3:37259)
0929 19:04:32.858235 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0929 19:04:32.858239 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 19:04:32.858253 (+ 14us) server_negotiation.cc:407] Connection header received
0929 19:04:32.858301 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:04:32.858303 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:04:32.858316 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:04:32.858397 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:04:32.858553 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:32.859230 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:32.860876 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:32.861336 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:32.861425 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:04:32.861504 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0929 19:04:32.861723 (+ 219us) server_negotiation.cc:299] Negotiation successful
0929 19:04:32.861765 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":69,"threads_started":1}
I20260929 19:04:32.861820 18398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:32.857588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42487 (local address 127.0.0.1:52618)
0929 19:04:32.857740 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0929 19:04:32.857787 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 19:04:32.857863 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:04:32.858093 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:04:32.858096 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:04:32.858098 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:04:32.858152 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:04:32.858153 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:04:32.859389 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:04:32.859390 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:04:32.860629 (+ 1239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:04:32.860631 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:04:32.861584 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:04:32.861586 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:04:32.861643 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:04:32.861647 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 19:04:32.861698 (+ 51us) client_negotiation.cc:240] Negotiation successful
0929 19:04:32.861743 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"mutex_wait_us":53,"thread_start_us":72,"threads_started":1}
I20260929 19:04:32.862038 18400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:32.857692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52618 (local address 127.0.98.2:42487)
0929 19:04:32.857881 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0929 19:04:32.857884 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 19:04:32.857895 (+ 11us) server_negotiation.cc:407] Connection header received
0929 19:04:32.857933 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:04:32.857935 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:04:32.857948 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:04:32.858018 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:04:32.858232 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:32.859322 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:32.860710 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:32.861421 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:32.861481 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:04:32.861550 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0929 19:04:32.861912 (+ 362us) server_negotiation.cc:299] Negotiation successful
0929 19:04:32.861958 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":74,"threads_started":1}
I20260929 19:04:32.862164 18399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:32.857783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37259 (local address 127.0.0.1:47648)
0929 19:04:32.858073 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0929 19:04:32.858082 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 19:04:32.858132 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:04:32.858447 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:04:32.858448 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:04:32.858449 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:04:32.858480 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:04:32.858481 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:04:32.859323 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:04:32.859324 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:04:32.860629 (+ 1305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:04:32.860631 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:04:32.861536 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:04:32.861537 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:04:32.861593 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:04:32.861595 (+ 2us) client_negotiation.cc:769] Sending connection context
0929 19:04:32.861640 (+ 45us) client_negotiation.cc:240] Negotiation successful
0929 19:04:32.861670 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"mutex_wait_us":33,"thread_start_us":67,"threads_started":1}
I20260929 19:04:32.862735 18062 tablet_service.cc:1467] Processing CreateTablet for tablet 1a16d258333d4250898ce8efd84071d0 (DEFAULT_TABLE table=test-table [id=c66563a7dab6462094d889774a64c13c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:04:32.862720 18193 tablet_service.cc:1467] Processing CreateTablet for tablet 1a16d258333d4250898ce8efd84071d0 (DEFAULT_TABLE table=test-table [id=c66563a7dab6462094d889774a64c13c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:04:32.862735 18324 tablet_service.cc:1467] Processing CreateTablet for tablet 1a16d258333d4250898ce8efd84071d0 (DEFAULT_TABLE table=test-table [id=c66563a7dab6462094d889774a64c13c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:04:32.863035 18062 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a16d258333d4250898ce8efd84071d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:04:32.863035 18324 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a16d258333d4250898ce8efd84071d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:04:32.863058 18193 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a16d258333d4250898ce8efd84071d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:04:32.865345 18404 tablet_bootstrap.cc:492] T 1a16d258333d4250898ce8efd84071d0 P 2c0a4d9e5e8c4b95b2d21e953a528e5f: Bootstrap starting.
I20260929 19:04:32.866036 18406 tablet_bootstrap.cc:492] T 1a16d258333d4250898ce8efd84071d0 P 1ff6e66969c94788954ea1bceb79cc18: Bootstrap starting.
I20260929 19:04:32.866240 18404 tablet_bootstrap.cc:654] T 1a16d258333d4250898ce8efd84071d0 P 2c0a4d9e5e8c4b95b2d21e953a528e5f: Neither blocks nor log segments found. Creating new log.
I20260929 19:04:32.866231 18405 tablet_bootstrap.cc:492] T 1a16d258333d4250898ce8efd84071d0 P 8cfb18cc973d4be39b49273acb1f40b5: Bootstrap starting.
I20260929 19:04:32.866582 18404 log.cc:824] T 1a16d258333d4250898ce8efd84071d0 P 2c0a4d9e5e8c4b95b2d21e953a528e5f: Log is configured to *not* fsync() on all Append() calls
I20260929 19:04:32.866843 18405 tablet_bootstrap.cc:654] T 1a16d258333d4250898ce8efd84071d0 P 8cfb18cc973d4be39b49273acb1f40b5: Neither blocks nor log segments found. Creating new log.
I20260929 19:04:32.867137 18405 log.cc:824] T 1a16d258333d4250898ce8efd84071d0 P 8cfb18cc973d4be39b49273acb1f40b5: Log is configured to *not* fsync() on all Append() calls
I20260929 19:04:32.867304 18404 tablet_bootstrap.cc:492] T 1a16d258333d4250898ce8efd84071d0 P 2c0a4d9e5e8c4b95b2d21e953a528e5f: No bootstrap required, opened a new log
I20260929 19:04:32.867390 18404 ts_tablet_manager.cc:1397] T 1a16d258333d4250898ce8efd84071d0 P 2c0a4d9e5e8c4b95b2d21e953a528e5f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 19:04:32.867823 18406 tablet_bootstrap.cc:654] T 1a16d258333d4250898ce8efd84071d0 P 1ff6e66969c94788954ea1bceb79cc18: Neither blocks nor log segments found. Creating new log.
I20260929 19:04:32.867844 18405 tablet_bootstrap.cc:492] T 1a16d258333d4250898ce8efd84071d0 P 8cfb18cc973d4be39b49273acb1f40b5: No bootstrap required, opened a new log
I20260929 19:04:32.867908 18405 ts_tablet_manager.cc:1397] T 1a16d258333d4250898ce8efd84071d0 P 8cfb18cc973d4be39b49273acb1f40b5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 19:04:32.868115 18406 log.cc:824] T 1a16d258333d4250898ce8efd84071d0 P 1ff6e66969c94788954ea1bceb79cc18: Log is configured to *not* fsync() on all Append() calls
I20260929 19:04:32.868745 18406 tablet_bootstrap.cc:492] T 1a16d258333d4250898ce8efd84071d0 P 1ff6e66969c94788954ea1bceb79cc18: No bootstrap required, opened a new log
I20260929 19:04:32.868973 18406 ts_tablet_manager.cc:1397] T 1a16d258333d4250898ce8efd84071d0 P 1ff6e66969c94788954ea1bceb79cc18: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260929 19:04:32.869951 18404 raft_consensus.cc:348] T 1a16d258333d4250898ce8efd84071d0 P 2c0a4d9e5e8c4b95b2d21e953a528e5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c0a4d9e5e8c4b95b2d21e953a528e5f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42487 } } peers { permanent_uuid: "8cfb18cc973d4be39b49273acb1f40b5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39269 } } peers { permanent_uuid: "1ff6e66969c94788954ea1bceb79cc18" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37259 } }
I20260929 19:04:32.870106 18404 raft_consensus.cc:374] T 1a16d258333d4250898ce8efd84071d0 P 2c0a4d9e5e8c4b95b2d21e953a528e5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:04:32.870144 18404 raft_consensus.cc:729] T 1a16d258333d4250898ce8efd84071d0 P 2c0a4d9e5e8c4b95b2d21e953a528e5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c0a4d9e5e8c4b95b2d21e953a528e5f, State: Initialized, Role: FOLLOWER
I20260929 19:04:32.870280 18404 consensus_queue.cc:260] T 1a16d258333d4250898ce8efd84071d0 P 2c0a4d9e5e8c4b95b2d21e953a528e5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c0a4d9e5e8c4b95b2d21e953a528e5f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42487 } } peers { permanent_uuid: "8cfb18cc973d4be39b49273acb1f40b5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39269 } } peers { permanent_uuid: "1ff6e66969c94788954ea1bceb79cc18" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37259 } }
I20260929 19:04:32.870407 18405 raft_consensus.cc:348] T 1a16d258333d4250898ce8efd84071d0 P 8cfb18cc973d4be39b49273acb1f40b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c0a4d9e5e8c4b95b2d21e953a528e5f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42487 } } peers { permanent_uuid: "8cfb18cc973d4be39b49273acb1f40b5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39269 } } peers { permanent_uuid: "1ff6e66969c94788954ea1bceb79cc18" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37259 } }
I20260929 19:04:32.870520 18404 ts_tablet_manager.cc:1428] T 1a16d258333d4250898ce8efd84071d0 P 2c0a4d9e5e8c4b95b2d21e953a528e5f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 19:04:32.870551 18405 raft_consensus.cc:374] T 1a16d258333d4250898ce8efd84071d0 P 8cfb18cc973d4be39b49273acb1f40b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:04:32.870625 18259 heartbeater.cc:503] Master 127.0.98.62:33229 was elected leader, sending a full tablet report...
I20260929 19:04:32.870625 18405 raft_consensus.cc:729] T 1a16d258333d4250898ce8efd84071d0 P 8cfb18cc973d4be39b49273acb1f40b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8cfb18cc973d4be39b49273acb1f40b5, State: Initialized, Role: FOLLOWER
I20260929 19:04:32.870733 18405 consensus_queue.cc:260] T 1a16d258333d4250898ce8efd84071d0 P 8cfb18cc973d4be39b49273acb1f40b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c0a4d9e5e8c4b95b2d21e953a528e5f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42487 } } peers { permanent_uuid: "8cfb18cc973d4be39b49273acb1f40b5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39269 } } peers { permanent_uuid: "1ff6e66969c94788954ea1bceb79cc18" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37259 } }
I20260929 19:04:32.870951 18405 ts_tablet_manager.cc:1428] T 1a16d258333d4250898ce8efd84071d0 P 8cfb18cc973d4be39b49273acb1f40b5: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 19:04:32.871095 18128 heartbeater.cc:503] Master 127.0.98.62:33229 was elected leader, sending a full tablet report...
I20260929 19:04:32.871176 18406 raft_consensus.cc:348] T 1a16d258333d4250898ce8efd84071d0 P 1ff6e66969c94788954ea1bceb79cc18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c0a4d9e5e8c4b95b2d21e953a528e5f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42487 } } peers { permanent_uuid: "8cfb18cc973d4be39b49273acb1f40b5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39269 } } peers { permanent_uuid: "1ff6e66969c94788954ea1bceb79cc18" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37259 } }
I20260929 19:04:32.871336 18406 raft_consensus.cc:374] T 1a16d258333d4250898ce8efd84071d0 P 1ff6e66969c94788954ea1bceb79cc18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:04:32.871373 18406 raft_consensus.cc:729] T 1a16d258333d4250898ce8efd84071d0 P 1ff6e66969c94788954ea1bceb79cc18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ff6e66969c94788954ea1bceb79cc18, State: Initialized, Role: FOLLOWER
I20260929 19:04:32.871469 18406 consensus_queue.cc:260] T 1a16d258333d4250898ce8efd84071d0 P 1ff6e66969c94788954ea1bceb79cc18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c0a4d9e5e8c4b95b2d21e953a528e5f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42487 } } peers { permanent_uuid: "8cfb18cc973d4be39b49273acb1f40b5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39269 } } peers { permanent_uuid: "1ff6e66969c94788954ea1bceb79cc18" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37259 } }
I20260929 19:04:32.871718 18406 ts_tablet_manager.cc:1428] T 1a16d258333d4250898ce8efd84071d0 P 1ff6e66969c94788954ea1bceb79cc18: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260929 19:04:32.871806 18390 heartbeater.cc:503] Master 127.0.98.62:33229 was elected leader, sending a full tablet report...
W20260929 19:04:32.968918 18260 tablet.cc:2369] T 1a16d258333d4250898ce8efd84071d0 P 2c0a4d9e5e8c4b95b2d21e953a528e5f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 19:04:33.084148 18391 tablet.cc:2369] T 1a16d258333d4250898ce8efd84071d0 P 1ff6e66969c94788954ea1bceb79cc18: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 19:04:33.103964 18129 tablet.cc:2369] T 1a16d258333d4250898ce8efd84071d0 P 8cfb18cc973d4be39b49273acb1f40b5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 19:04:33.113605 18411 raft_consensus.cc:482] T 1a16d258333d4250898ce8efd84071d0 P 8cfb18cc973d4be39b49273acb1f40b5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 19:04:33.113685 18411 raft_consensus.cc:504] T 1a16d258333d4250898ce8efd84071d0 P 8cfb18cc973d4be39b49273acb1f40b5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c0a4d9e5e8c4b95b2d21e953a528e5f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42487 } } peers { permanent_uuid: "8cfb18cc973d4be39b49273acb1f40b5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39269 } } peers { permanent_uuid: "1ff6e66969c94788954ea1bceb79cc18" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37259 } }
I20260929 19:04:33.114037 18411 leader_election.cc:288] T 1a16d258333d4250898ce8efd84071d0 P 8cfb18cc973d4be39b49273acb1f40b5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2c0a4d9e5e8c4b95b2d21e953a528e5f (127.0.98.2:42487), 1ff6e66969c94788954ea1bceb79cc18 (127.0.98.3:37259)
I20260929 19:04:33.116864 18403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:33.114246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43487 (local address 127.0.98.3:37259)
0929 19:04:33.114285 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0929 19:04:33.114287 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:04:33.114354 (+ 67us) server_negotiation.cc:407] Connection header received
0929 19:04:33.114404 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:04:33.114405 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:04:33.114414 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:04:33.114481 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:04:33.114647 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:33.115187 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:33.116140 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:33.116519 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:33.116600 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:04:33.116610 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0929 19:04:33.116749 (+ 139us) server_negotiation.cc:299] Negotiation successful
0929 19:04:33.116784 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260929 19:04:33.116890 18414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:33.114185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37259 (local address 127.0.98.1:43487)
0929 19:04:33.114282 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0929 19:04:33.114291 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 19:04:33.114348 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:04:33.114485 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:04:33.114486 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:04:33.114487 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:04:33.114574 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:04:33.114576 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:04:33.115260 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:04:33.115261 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:04:33.116025 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:04:33.116045 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:04:33.116599 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:04:33.116600 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:04:33.116676 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:04:33.116679 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 19:04:33.116744 (+ 65us) client_negotiation.cc:240] Negotiation successful
0929 19:04:33.116782 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86,"thread_start_us":46,"threads_started":1}
I20260929 19:04:33.117096 18400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:33.114231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37167 (local address 127.0.98.2:42487)
0929 19:04:33.114271 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0929 19:04:33.114273 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:04:33.114355 (+ 82us) server_negotiation.cc:407] Connection header received
0929 19:04:33.114424 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:04:33.114424 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:04:33.114432 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:04:33.114493 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:04:33.114675 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:33.115221 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:33.116275 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:33.116691 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:33.116769 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:04:33.116777 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0929 19:04:33.116981 (+ 204us) server_negotiation.cc:299] Negotiation successful
0929 19:04:33.117031 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260929 19:04:33.117190 18344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a16d258333d4250898ce8efd84071d0" candidate_uuid: "8cfb18cc973d4be39b49273acb1f40b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ff6e66969c94788954ea1bceb79cc18" is_pre_election: true
I20260929 19:04:33.117308 18413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:33.114177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42487 (local address 127.0.98.1:37167)
0929 19:04:33.114279 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0929 19:04:33.114289 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 19:04:33.114370 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:04:33.114508 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:04:33.114509 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:04:33.114510 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:04:33.114600 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:04:33.114602 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:04:33.115294 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:04:33.115296 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:04:33.116191 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:04:33.116193 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:04:33.116771 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:04:33.116772 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:04:33.116883 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:04:33.116888 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 19:04:33.117185 (+ 297us) client_negotiation.cc:240] Negotiation successful
0929 19:04:33.117229 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87,"thread_start_us":51,"threads_started":1}
I20260929 19:04:33.117340 18344 raft_consensus.cc:2454] T 1a16d258333d4250898ce8efd84071d0 P 1ff6e66969c94788954ea1bceb79cc18 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8cfb18cc973d4be39b49273acb1f40b5 in term 0.
I20260929 19:04:33.117599 18016 leader_election.cc:302] T 1a16d258333d4250898ce8efd84071d0 P 8cfb18cc973d4be39b49273acb1f40b5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ff6e66969c94788954ea1bceb79cc18, 8cfb18cc973d4be39b49273acb1f40b5; no voters:
I20260929 19:04:33.117599 18213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a16d258333d4250898ce8efd84071d0" candidate_uuid: "8cfb18cc973d4be39b49273acb1f40b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c0a4d9e5e8c4b95b2d21e953a528e5f" is_pre_election: true
I20260929 19:04:33.117729 18213 raft_consensus.cc:2454] T 1a16d258333d4250898ce8efd84071d0 P 2c0a4d9e5e8c4b95b2d21e953a528e5f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8cfb18cc973d4be39b49273acb1f40b5 in term 0.
I20260929 19:04:33.117735 18411 raft_consensus.cc:2781] T 1a16d258333d4250898ce8efd84071d0 P 8cfb18cc973d4be39b49273acb1f40b5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 19:04:33.117794 18411 raft_consensus.cc:482] T 1a16d258333d4250898ce8efd84071d0 P 8cfb18cc973d4be39b49273acb1f40b5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 19:04:33.117832 18411 raft_consensus.cc:3037] T 1a16d258333d4250898ce8efd84071d0 P 8cfb18cc973d4be39b49273acb1f40b5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:04:33.118577 18411 raft_consensus.cc:504] T 1a16d258333d4250898ce8efd84071d0 P 8cfb18cc973d4be39b49273acb1f40b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c0a4d9e5e8c4b95b2d21e953a528e5f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42487 } } peers { permanent_uuid: "8cfb18cc973d4be39b49273acb1f40b5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39269 } } peers { permanent_uuid: "1ff6e66969c94788954ea1bceb79cc18" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37259 } }
I20260929 19:04:33.118708 18411 leader_election.cc:288] T 1a16d258333d4250898ce8efd84071d0 P 8cfb18cc973d4be39b49273acb1f40b5 [CANDIDATE]: Term 1 election: Requested vote from peers 2c0a4d9e5e8c4b95b2d21e953a528e5f (127.0.98.2:42487), 1ff6e66969c94788954ea1bceb79cc18 (127.0.98.3:37259)
I20260929 19:04:33.118855 18213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a16d258333d4250898ce8efd84071d0" candidate_uuid: "8cfb18cc973d4be39b49273acb1f40b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c0a4d9e5e8c4b95b2d21e953a528e5f"
I20260929 19:04:33.118871 18344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a16d258333d4250898ce8efd84071d0" candidate_uuid: "8cfb18cc973d4be39b49273acb1f40b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ff6e66969c94788954ea1bceb79cc18"
I20260929 19:04:33.118934 18344 raft_consensus.cc:3037] T 1a16d258333d4250898ce8efd84071d0 P 1ff6e66969c94788954ea1bceb79cc18 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:04:33.118934 18213 raft_consensus.cc:3037] T 1a16d258333d4250898ce8efd84071d0 P 2c0a4d9e5e8c4b95b2d21e953a528e5f [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:04:33.119828 18344 raft_consensus.cc:2454] T 1a16d258333d4250898ce8efd84071d0 P 1ff6e66969c94788954ea1bceb79cc18 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8cfb18cc973d4be39b49273acb1f40b5 in term 1.
I20260929 19:04:33.119828 18213 raft_consensus.cc:2454] T 1a16d258333d4250898ce8efd84071d0 P 2c0a4d9e5e8c4b95b2d21e953a528e5f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8cfb18cc973d4be39b49273acb1f40b5 in term 1.
I20260929 19:04:33.119993 18016 leader_election.cc:302] T 1a16d258333d4250898ce8efd84071d0 P 8cfb18cc973d4be39b49273acb1f40b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ff6e66969c94788954ea1bceb79cc18, 8cfb18cc973d4be39b49273acb1f40b5; no voters:
I20260929 19:04:33.120086 18411 raft_consensus.cc:2781] T 1a16d258333d4250898ce8efd84071d0 P 8cfb18cc973d4be39b49273acb1f40b5 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:04:33.120208 18411 raft_consensus.cc:686] T 1a16d258333d4250898ce8efd84071d0 P 8cfb18cc973d4be39b49273acb1f40b5 [term 1 LEADER]: Becoming Leader. State: Replica: 8cfb18cc973d4be39b49273acb1f40b5, State: Running, Role: LEADER
I20260929 19:04:33.120304 18411 consensus_queue.cc:237] T 1a16d258333d4250898ce8efd84071d0 P 8cfb18cc973d4be39b49273acb1f40b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c0a4d9e5e8c4b95b2d21e953a528e5f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42487 } } peers { permanent_uuid: "8cfb18cc973d4be39b49273acb1f40b5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39269 } } peers { permanent_uuid: "1ff6e66969c94788954ea1bceb79cc18" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37259 } }
I20260929 19:04:33.121070 17940 catalog_manager.cc:5495] T 1a16d258333d4250898ce8efd84071d0 P 8cfb18cc973d4be39b49273acb1f40b5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8cfb18cc973d4be39b49273acb1f40b5 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "8cfb18cc973d4be39b49273acb1f40b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c0a4d9e5e8c4b95b2d21e953a528e5f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42487 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8cfb18cc973d4be39b49273acb1f40b5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39269 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1ff6e66969c94788954ea1bceb79cc18" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37259 } health_report { overall_health: UNKNOWN } } }
I20260929 19:04:33.141799 18402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:33.139542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58220 (local address 127.0.98.1:39269)
0929 19:04:33.139590 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0929 19:04:33.139592 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:04:33.139615 (+ 23us) server_negotiation.cc:407] Connection header received
0929 19:04:33.139651 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:04:33.139652 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:04:33.139666 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:04:33.139736 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 19:04:33.139851 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:33.140268 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:33.141068 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:33.141249 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:33.141339 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:04:33.141518 (+ 179us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 19:04:33.141604 (+ 86us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 19:04:33.141694 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0929 19:04:33.141713 (+ 19us) server_negotiation.cc:299] Negotiation successful
0929 19:04:33.141731 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260929 19:04:33.143383 18213 raft_consensus.cc:1261] T 1a16d258333d4250898ce8efd84071d0 P 2c0a4d9e5e8c4b95b2d21e953a528e5f [term 1 FOLLOWER]: Refusing update from remote peer 8cfb18cc973d4be39b49273acb1f40b5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 19:04:33.143393 18344 raft_consensus.cc:1261] T 1a16d258333d4250898ce8efd84071d0 P 1ff6e66969c94788954ea1bceb79cc18 [term 1 FOLLOWER]: Refusing update from remote peer 8cfb18cc973d4be39b49273acb1f40b5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 19:04:33.143728 18411 consensus_queue.cc:1035] T 1a16d258333d4250898ce8efd84071d0 P 8cfb18cc973d4be39b49273acb1f40b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ff6e66969c94788954ea1bceb79cc18" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 19:04:33.143860 18415 consensus_queue.cc:1035] T 1a16d258333d4250898ce8efd84071d0 P 8cfb18cc973d4be39b49273acb1f40b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c0a4d9e5e8c4b95b2d21e953a528e5f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42487 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 19:04:33.146116 18419 mvcc.cc:205] Tried to move back new op lower bound from 7334742725192953856 to 7334742725101445120. Current Snapshot: MvccSnapshot[applied={T|T < 7334742725192953856}]
I20260929 19:04:33.147085 18418 mvcc.cc:205] Tried to move back new op lower bound from 7334742725192953856 to 7334742725101445120. Current Snapshot: MvccSnapshot[applied={T|T < 7334742725192953856}]
I20260929 19:04:33.147236 18420 mvcc.cc:205] Tried to move back new op lower bound from 7334742725192953856 to 7334742725101445120. Current Snapshot: MvccSnapshot[applied={T|T < 7334742725192953856 or (T in {7334742725192953856})}]
I20260929 19:04:33.149075 17939 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33770:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 19:04:33.149180 17939 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33770:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 19:04:33.149803 17939 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 254e493dfb484453b093430efb0bd798: Sending DeleteTablet for 3 replicas of tablet 1a16d258333d4250898ce8efd84071d0
I20260929 19:04:33.150184 18193 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a16d258333d4250898ce8efd84071d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:04:33 UTC) from {username='slave'} at 127.0.0.1:52618
I20260929 19:04:33.150388 18062 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a16d258333d4250898ce8efd84071d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:04:33 UTC) from {username='slave'} at 127.0.0.1:58204
I20260929 19:04:33.150408 18428 tablet_replica.cc:330] T 1a16d258333d4250898ce8efd84071d0 P 2c0a4d9e5e8c4b95b2d21e953a528e5f: stopping tablet replica
I20260929 19:04:33.150426 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUbnLHE/build/release/bin/kudu with pid 18000
I20260929 19:04:33.150516 18428 raft_consensus.cc:2229] T 1a16d258333d4250898ce8efd84071d0 P 2c0a4d9e5e8c4b95b2d21e953a528e5f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 19:04:33.150604 18428 raft_consensus.cc:2258] T 1a16d258333d4250898ce8efd84071d0 P 2c0a4d9e5e8c4b95b2d21e953a528e5f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 19:04:33.151053 18324 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a16d258333d4250898ce8efd84071d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:04:33 UTC) from {username='slave'} at 127.0.0.1:47648
I20260929 19:04:33.151268 18429 tablet_replica.cc:330] T 1a16d258333d4250898ce8efd84071d0 P 1ff6e66969c94788954ea1bceb79cc18: stopping tablet replica
I20260929 19:04:33.151364 18429 raft_consensus.cc:2229] T 1a16d258333d4250898ce8efd84071d0 P 1ff6e66969c94788954ea1bceb79cc18 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 19:04:33.151460 18429 raft_consensus.cc:2258] T 1a16d258333d4250898ce8efd84071d0 P 1ff6e66969c94788954ea1bceb79cc18 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 19:04:33.151819 18429 ts_tablet_manager.cc:1905] T 1a16d258333d4250898ce8efd84071d0 P 1ff6e66969c94788954ea1bceb79cc18: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 19:04:33.152040 18428 ts_tablet_manager.cc:1905] T 1a16d258333d4250898ce8efd84071d0 P 2c0a4d9e5e8c4b95b2d21e953a528e5f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 19:04:33.153538 18428 ts_tablet_manager.cc:1918] T 1a16d258333d4250898ce8efd84071d0 P 2c0a4d9e5e8c4b95b2d21e953a528e5f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 19:04:33.153607 18428 log.cc:1196] T 1a16d258333d4250898ce8efd84071d0 P 2c0a4d9e5e8c4b95b2d21e953a528e5f: Deleting WAL directory at /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708447806542-392-0/minicluster-data/ts-1/wal/wals/1a16d258333d4250898ce8efd84071d0
I20260929 19:04:33.153750 18429 ts_tablet_manager.cc:1918] T 1a16d258333d4250898ce8efd84071d0 P 1ff6e66969c94788954ea1bceb79cc18: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 19:04:33.153826 18429 log.cc:1196] T 1a16d258333d4250898ce8efd84071d0 P 1ff6e66969c94788954ea1bceb79cc18: Deleting WAL directory at /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790708447806542-392-0/minicluster-data/ts-2/wal/wals/1a16d258333d4250898ce8efd84071d0
I20260929 19:04:33.153853 18428 ts_tablet_manager.cc:1939] T 1a16d258333d4250898ce8efd84071d0 P 2c0a4d9e5e8c4b95b2d21e953a528e5f: Deleting consensus metadata
I20260929 19:04:33.154033 18429 ts_tablet_manager.cc:1939] T 1a16d258333d4250898ce8efd84071d0 P 1ff6e66969c94788954ea1bceb79cc18: Deleting consensus metadata
I20260929 19:04:33.154237 17926 catalog_manager.cc:4841] TS 2c0a4d9e5e8c4b95b2d21e953a528e5f (127.0.98.2:42487): tablet 1a16d258333d4250898ce8efd84071d0 (table test-table [id=c66563a7dab6462094d889774a64c13c]) successfully deleted
I20260929 19:04:33.154381 17925 catalog_manager.cc:4841] TS 1ff6e66969c94788954ea1bceb79cc18 (127.0.98.3:37259): tablet 1a16d258333d4250898ce8efd84071d0 (table test-table [id=c66563a7dab6462094d889774a64c13c]) successfully deleted
W20260929 19:04:33.156826 17925 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:39269 (error 108)
I20260929 19:04:33.157085 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUbnLHE/build/release/bin/kudu with pid 18131
I20260929 19:04:33.157671 18398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:33.157123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39269 (local address 127.0.0.1:58236)
0929 19:04:33.157590 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0929 19:04:33.157636 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39269: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":446}
W20260929 19:04:33.157866 17925 catalog_manager.cc:4584] TS 8cfb18cc973d4be39b49273acb1f40b5 (127.0.98.1:39269): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1a16d258333d4250898ce8efd84071d0: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39269: connect: Connection refused (error 111)
I20260929 19:04:33.161895 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUbnLHE/build/release/bin/kudu with pid 18262
I20260929 19:04:33.166137 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUbnLHE/build/release/bin/kudu with pid 17909
2026-09-29T19:04:33Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (834 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708447806542-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 19:04:33 dist-test-slave-lqkv krb5kdc[18433](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 19:04:33 dist-test-slave-lqkv krb5kdc[18433](info): set up 2 sockets
Sep 29 19:04:33 dist-test-slave-lqkv krb5kdc[18433](info): commencing operation
krb5kdc: starting...
W20260929 19:04:34.124191 392 mini_kdc.cc:121] Time spent starting KDC: real 0.939s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708447806542-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708447806542-392-0/krb5kdc/test-user.keytab.
Sep 29 19:04:34 dist-test-slave-lqkv krb5kdc[18433](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708674, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T19:04:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T19:04:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708447806542-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708447806542-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 19:04:34.227475 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUbnLHE/build/release/bin/kudu
/tmp/dist-test-taskUbnLHE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708447806542-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708447806542-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708447806542-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708447806542-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45477
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708447806542-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45477
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708447806542-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708447806542-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708447806542-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:04:34.298486 18446 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:04:34.298637 18446 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:04:34.298655 18446 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:04:34.299659 18446 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 19:04:34.299695 18446 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:04:34.299708 18446 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:04:34.299741 18446 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 19:04:34.299767 18446 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 19:04:34.301154 18446 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708447806542-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708447806542-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45477
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708447806542-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45477
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708447806542-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18446
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708447806542-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 18:09:13 UTC on ccycloud.quasar-hpkalf.root.comops.site
build id 4680913
I20260929 19:04:34.301373 18446 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:04:34.301573 18446 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:04:34.305609 18454 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:04:34.305613 18452 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:04:34.305822 18446 server_base.cc:1034] running on GCE node
W20260929 19:04:34.305652 18451 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:04:34.306108 18446 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:04:34.306388 18446 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:04:34.307521 18446 hybrid_clock.cc:648] HybridClock initialized: now 1790708674307511 us; error 26 us; skew 500 ppm
Sep 29 19:04:34 dist-test-slave-lqkv krb5kdc[18433](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708674, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:04:34.309897 18446 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 19:04:34.310323 18446 webserver.cc:460] Webserver started at http://127.0.98.62:39759/ using document root <none> and password file <none>
I20260929 19:04:34.310526 18446 fs_manager.cc:359] Metadata directory not provided
I20260929 19:04:34.310567 18446 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:04:34.310707 18446 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:04:34.311623 18446 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708447806542-392-0/minicluster-data/master-0/data/instance:
uuid: "9325793030b244efb80f847990e12a95"
format_stamp: "Formatted at 2026-09-29 19:04:34 on dist-test-slave-lqkv"
server_key: "9821964a5be31d1f29058c9d6941226b"
server_key_iv: "e41f0a556f73a7b8a580d45f8b65a6d6"
server_key_version: "encryptionkey@0"
I20260929 19:04:34.311946 18446 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708447806542-392-0/minicluster-data/master-0/wal/instance:
uuid: "9325793030b244efb80f847990e12a95"
format_stamp: "Formatted at 2026-09-29 19:04:34 on dist-test-slave-lqkv"
server_key: "9821964a5be31d1f29058c9d6941226b"
server_key_iv: "e41f0a556f73a7b8a580d45f8b65a6d6"
server_key_version: "encryptionkey@0"
I20260929 19:04:34.313184 18446 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 19:04:34.313800 18461 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:04:34.313941 18446 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 19:04:34.314025 18446 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708447806542-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708447806542-392-0/minicluster-data/master-0/wal
uuid: "9325793030b244efb80f847990e12a95"
format_stamp: "Formatted at 2026-09-29 19:04:34 on dist-test-slave-lqkv"
server_key: "9821964a5be31d1f29058c9d6941226b"
server_key_iv: "e41f0a556f73a7b8a580d45f8b65a6d6"
server_key_version: "encryptionkey@0"
I20260929 19:04:34.314085 18446 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708447806542-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708447806542-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708447806542-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:04:34.331422 18446 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:04:34.333443 18446 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:04:34.333575 18446 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:04:34.337122 18446 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45477
I20260929 19:04:34.337149 18513 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45477 every 8 connection(s)
I20260929 19:04:34.337487 18446 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708447806542-392-0/minicluster-data/master-0/data/info.pb
I20260929 19:04:34.338153 18514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:04:34.340052 18514 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9325793030b244efb80f847990e12a95: Bootstrap starting.
I20260929 19:04:34.340533 18514 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9325793030b244efb80f847990e12a95: Neither blocks nor log segments found. Creating new log.
I20260929 19:04:34.340749 18514 log.cc:824] T 00000000000000000000000000000000 P 9325793030b244efb80f847990e12a95: Log is configured to *not* fsync() on all Append() calls
I20260929 19:04:34.341288 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUbnLHE/build/release/bin/kudu as pid 18446
I20260929 19:04:34.341404 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708447806542-392-0/minicluster-data/master-0/wal/instance
I20260929 19:04:34.341451 18514 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9325793030b244efb80f847990e12a95: No bootstrap required, opened a new log
I20260929 19:04:34.341630 392 external_mini_cluster.cc:1428] Setting key b20bbc6071c93735032fa6b7436b0841
I20260929 19:04:34.343133 18514 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9325793030b244efb80f847990e12a95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9325793030b244efb80f847990e12a95" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45477 } }
I20260929 19:04:34.343267 18514 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9325793030b244efb80f847990e12a95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:04:34.343319 18514 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9325793030b244efb80f847990e12a95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9325793030b244efb80f847990e12a95, State: Initialized, Role: FOLLOWER
I20260929 19:04:34.343434 18514 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9325793030b244efb80f847990e12a95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9325793030b244efb80f847990e12a95" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45477 } }
I20260929 19:04:34.343510 18514 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9325793030b244efb80f847990e12a95 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 19:04:34.343544 18514 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9325793030b244efb80f847990e12a95 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 19:04:34.343587 18514 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9325793030b244efb80f847990e12a95 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:04:34.344360 18514 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9325793030b244efb80f847990e12a95 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9325793030b244efb80f847990e12a95" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45477 } }
I20260929 19:04:34.344437 18514 leader_election.cc:302] T 00000000000000000000000000000000 P 9325793030b244efb80f847990e12a95 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9325793030b244efb80f847990e12a95; no voters:
I20260929 19:04:34.344606 18514 leader_election.cc:288] T 00000000000000000000000000000000 P 9325793030b244efb80f847990e12a95 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 19:04:34.344623 18519 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9325793030b244efb80f847990e12a95 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:04:34.344740 18519 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9325793030b244efb80f847990e12a95 [term 1 LEADER]: Becoming Leader. State: Replica: 9325793030b244efb80f847990e12a95, State: Running, Role: LEADER
I20260929 19:04:34.344875 18514 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9325793030b244efb80f847990e12a95 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 19:04:34.344897 18519 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9325793030b244efb80f847990e12a95 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9325793030b244efb80f847990e12a95" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45477 } }
I20260929 19:04:34.345299 18521 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9325793030b244efb80f847990e12a95 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9325793030b244efb80f847990e12a95" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9325793030b244efb80f847990e12a95" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45477 } } }
I20260929 19:04:34.345311 18520 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9325793030b244efb80f847990e12a95 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9325793030b244efb80f847990e12a95. Latest consensus state: current_term: 1 leader_uuid: "9325793030b244efb80f847990e12a95" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9325793030b244efb80f847990e12a95" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45477 } } }
I20260929 19:04:34.345475 18520 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9325793030b244efb80f847990e12a95 [sys.catalog]: This master's current role is: LEADER
I20260929 19:04:34.345662 18521 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9325793030b244efb80f847990e12a95 [sys.catalog]: This master's current role is: LEADER
I20260929 19:04:34.345785 18528 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 19:04:34.346246 18528 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 19:04:34.347975 18528 catalog_manager.cc:1347] Generated new cluster ID: f72166a5395042f7b628875b300ed129
I20260929 19:04:34.348068 18528 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 29 19:04:34 dist-test-slave-lqkv krb5kdc[18433](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708674, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 19:04:34.352909 18518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:34.342014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33680 (local address 127.0.98.62:45477)
0929 19:04:34.342165 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0929 19:04:34.342171 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 19:04:34.342195 (+ 24us) server_negotiation.cc:407] Connection header received
0929 19:04:34.343111 (+ 916us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:04:34.343115 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:04:34.343150 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:04:34.343239 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:04:34.343950 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:34.344982 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:34.346107 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:34.346765 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:34.347152 (+ 387us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:04:34.349468 (+ 2316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:04:34.349471 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:04:34.349471 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:04:34.349481 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:04:34.351237 (+ 1756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:04:34.351671 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:04:34.351674 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:04:34.351674 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:04:34.351786 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:04:34.351992 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:04:34.351993 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:04:34.351993 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:04:34.352242 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:04:34.352381 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0929 19:04:34.352498 (+ 117us) server_negotiation.cc:299] Negotiation successful
0929 19:04:34.352703 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":68,"threads_started":1}
I20260929 19:04:34.354409 18528 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 19:04:34.354806 18528 catalog_manager.cc:1504] Loading token signing keys...
I20260929 19:04:34.361337 18528 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9325793030b244efb80f847990e12a95: Generated new TSK 0
I20260929 19:04:34.361496 18528 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708447806542-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708447806542-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260929 19:04:34.385151 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUbnLHE/build/release/bin/kudu
/tmp/dist-test-taskUbnLHE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708447806542-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708447806542-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708447806542-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708447806542-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45477
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708447806542-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708447806542-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708447806542-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708447806542-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:04:34.457857 18540 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:04:34.458048 18540 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:04:34.458078 18540 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:04:34.459131 18540 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:04:34.459187 18540 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:04:34.459225 18540 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 19:04:34.460496 18540 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708447806542-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708447806542-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708447806542-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708447806542-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45477
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18540
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708447806542-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 18:09:13 UTC on ccycloud.quasar-hpkalf.root.comops.site
build id 4680913
I20260929 19:04:34.460719 18540 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:04:34.460968 18540 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:04:34.465349 18545 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:04:34.465332 18546 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:04:34.465435 18548 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:04:34.465437 18540 server_base.cc:1034] running on GCE node
I20260929 19:04:34.465749 18540 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:04:34.465967 18540 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:04:34.467099 18540 hybrid_clock.cc:648] HybridClock initialized: now 1790708674467078 us; error 33 us; skew 500 ppm
Sep 29 19:04:34 dist-test-slave-lqkv krb5kdc[18433](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708674, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:04:34.472472 18540 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 19:04:34.472949 18540 webserver.cc:460] Webserver started at http://127.0.98.1:34871/ using document root <none> and password file <none>
I20260929 19:04:34.473156 18540 fs_manager.cc:359] Metadata directory not provided
I20260929 19:04:34.473217 18540 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:04:34.473341 18540 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:04:34.474331 18540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708447806542-392-0/minicluster-data/ts-0/data/instance:
uuid: "2cd8d48de66142be8a36e8278b01ed16"
format_stamp: "Formatted at 2026-09-29 19:04:34 on dist-test-slave-lqkv"
server_key: "2ba69ea0e99a5a9f33f5834d6d15f1b7"
server_key_iv: "40faf0d79870fe4f823cfee726f9e18c"
server_key_version: "encryptionkey@0"
I20260929 19:04:34.474646 18540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708447806542-392-0/minicluster-data/ts-0/wal/instance:
uuid: "2cd8d48de66142be8a36e8278b01ed16"
format_stamp: "Formatted at 2026-09-29 19:04:34 on dist-test-slave-lqkv"
server_key: "2ba69ea0e99a5a9f33f5834d6d15f1b7"
server_key_iv: "40faf0d79870fe4f823cfee726f9e18c"
server_key_version: "encryptionkey@0"
I20260929 19:04:34.475845 18540 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 19:04:34.476526 18555 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:04:34.476683 18540 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 19:04:34.476774 18540 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708447806542-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708447806542-392-0/minicluster-data/ts-0/wal
uuid: "2cd8d48de66142be8a36e8278b01ed16"
format_stamp: "Formatted at 2026-09-29 19:04:34 on dist-test-slave-lqkv"
server_key: "2ba69ea0e99a5a9f33f5834d6d15f1b7"
server_key_iv: "40faf0d79870fe4f823cfee726f9e18c"
server_key_version: "encryptionkey@0"
I20260929 19:04:34.476874 18540 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708447806542-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708447806542-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708447806542-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:04:34.495258 18540 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:04:34.497586 18540 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:04:34.497742 18540 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:04:34.497946 18540 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:04:34.498217 18540 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:04:34.498265 18540 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:04:34.498310 18540 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:04:34.498338 18540 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:04:34.503861 18540 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43765
I20260929 19:04:34.503908 18668 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43765 every 8 connection(s)
I20260929 19:04:34.504177 18540 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708447806542-392-0/minicluster-data/ts-0/data/info.pb
I20260929 19:04:34.509285 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUbnLHE/build/release/bin/kudu as pid 18540
I20260929 19:04:34.509413 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708447806542-392-0/minicluster-data/ts-0/wal/instance
I20260929 19:04:34.509599 392 external_mini_cluster.cc:1428] Setting key 018cb48ac3b070b519dfa967473fdb9d
Sep 29 19:04:34 dist-test-slave-lqkv krb5kdc[18433](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708674, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 19:04:34.513734 18671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:34.504763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45477 (local address 127.0.98.1:56083)
0929 19:04:34.504973 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0929 19:04:34.505005 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0929 19:04:34.505877 (+ 872us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:04:34.506134 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:04:34.506138 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:04:34.506396 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:04:34.506563 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:04:34.506567 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:04:34.507277 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:04:34.507278 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:04:34.508062 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:04:34.508064 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:04:34.508499 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:04:34.508500 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:04:34.508619 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:04:34.509090 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:04:34.509098 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:04:34.510583 (+ 1485us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:04:34.512465 (+ 1882us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:04:34.512471 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:04:34.512472 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:04:34.512679 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:04:34.512899 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:04:34.512900 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:04:34.512900 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:04:34.512992 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:04:34.513285 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:04:34.513290 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:04:34.513399 (+ 109us) client_negotiation.cc:769] Sending connection context
0929 19:04:34.513477 (+ 78us) client_negotiation.cc:240] Negotiation successful
0929 19:04:34.513594 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":115,"threads_started":1}
I20260929 19:04:34.513763 18518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:34.504787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56083 (local address 127.0.98.62:45477)
0929 19:04:34.504863 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0929 19:04:34.504865 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:04:34.505082 (+ 217us) server_negotiation.cc:407] Connection header received
0929 19:04:34.505990 (+ 908us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:04:34.505992 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:04:34.506005 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:04:34.506048 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:04:34.506661 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:34.507174 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:34.508178 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:34.508406 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:34.508488 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:04:34.510677 (+ 2189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:04:34.510678 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:04:34.510679 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:04:34.510688 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:04:34.512379 (+ 1691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:04:34.512753 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:04:34.512754 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:04:34.512755 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:04:34.512837 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:04:34.513057 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:04:34.513059 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:04:34.513060 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:04:34.513213 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:04:34.513283 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 19:04:34.513541 (+ 258us) server_negotiation.cc:299] Negotiation successful
0929 19:04:34.513636 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 19:04:34.514173 18669 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45477
I20260929 19:04:34.514299 18669 heartbeater.cc:464] Registering TS with master...
I20260929 19:04:34.514474 18669 heartbeater.cc:511] Master 127.0.98.62:45477 requested a full tablet report, sending...
I20260929 19:04:34.514961 18478 ts_manager.cc:194] Registered new tserver with Master: 2cd8d48de66142be8a36e8278b01ed16 (127.0.98.1:43765)
I20260929 19:04:34.515556 18478 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:56083
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708447806542-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708447806542-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260929 19:04:34.532922 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUbnLHE/build/release/bin/kudu
/tmp/dist-test-taskUbnLHE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708447806542-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708447806542-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708447806542-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708447806542-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45477
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708447806542-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708447806542-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708447806542-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708447806542-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:04:34.623951 18674 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:04:34.624155 18674 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:04:34.624192 18674 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:04:34.625367 18674 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:04:34.625422 18674 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:04:34.625474 18674 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 19:04:34.626984 18674 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708447806542-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708447806542-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708447806542-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708447806542-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45477
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18674
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708447806542-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 18:09:13 UTC on ccycloud.quasar-hpkalf.root.comops.site
build id 4680913
I20260929 19:04:34.627254 18674 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:04:34.627476 18674 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:04:34.631877 18679 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:04:34.631907 18680 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:04:34.631883 18682 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:04:34.632154 18674 server_base.cc:1034] running on GCE node
I20260929 19:04:34.632335 18674 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:04:34.632495 18674 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:04:34.633648 18674 hybrid_clock.cc:648] HybridClock initialized: now 1790708674633634 us; error 24 us; skew 500 ppm
Sep 29 19:04:34 dist-test-slave-lqkv krb5kdc[18433](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708674, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:04:34.635875 18674 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 19:04:34.636271 18674 webserver.cc:460] Webserver started at http://127.0.98.2:41187/ using document root <none> and password file <none>
I20260929 19:04:34.636475 18674 fs_manager.cc:359] Metadata directory not provided
I20260929 19:04:34.636538 18674 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:04:34.636677 18674 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:04:34.637671 18674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708447806542-392-0/minicluster-data/ts-1/data/instance:
uuid: "f701558d57854f36b7bc20434382737c"
format_stamp: "Formatted at 2026-09-29 19:04:34 on dist-test-slave-lqkv"
server_key: "bd82a118cc177bd2d3f9d54200f9d8a6"
server_key_iv: "e461ee835a1fea1403be08bca0e01823"
server_key_version: "encryptionkey@0"
I20260929 19:04:34.638013 18674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708447806542-392-0/minicluster-data/ts-1/wal/instance:
uuid: "f701558d57854f36b7bc20434382737c"
format_stamp: "Formatted at 2026-09-29 19:04:34 on dist-test-slave-lqkv"
server_key: "bd82a118cc177bd2d3f9d54200f9d8a6"
server_key_iv: "e461ee835a1fea1403be08bca0e01823"
server_key_version: "encryptionkey@0"
I20260929 19:04:34.639168 18674 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 19:04:34.639756 18689 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:04:34.639897 18674 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 19:04:34.639981 18674 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708447806542-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708447806542-392-0/minicluster-data/ts-1/wal
uuid: "f701558d57854f36b7bc20434382737c"
format_stamp: "Formatted at 2026-09-29 19:04:34 on dist-test-slave-lqkv"
server_key: "bd82a118cc177bd2d3f9d54200f9d8a6"
server_key_iv: "e461ee835a1fea1403be08bca0e01823"
server_key_version: "encryptionkey@0"
I20260929 19:04:34.640054 18674 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708447806542-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708447806542-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708447806542-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:04:34.652984 18674 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:04:34.654502 18674 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:04:34.654633 18674 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:04:34.654841 18674 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:04:34.655105 18674 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:04:34.655154 18674 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:04:34.655190 18674 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:04:34.655215 18674 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:04:34.660336 18674 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37553
I20260929 19:04:34.660403 18802 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37553 every 8 connection(s)
I20260929 19:04:34.660671 18674 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708447806542-392-0/minicluster-data/ts-1/data/info.pb
Sep 29 19:04:34 dist-test-slave-lqkv krb5kdc[18433](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708674, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 19:04:34.667102 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUbnLHE/build/release/bin/kudu as pid 18674
I20260929 19:04:34.667251 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708447806542-392-0/minicluster-data/ts-1/wal/instance
I20260929 19:04:34.667457 392 external_mini_cluster.cc:1428] Setting key 97a88b32e63d51f8f9d3ff682ad3f28c
I20260929 19:04:34.669173 18518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:34.661308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33463 (local address 127.0.98.62:45477)
0929 19:04:34.661375 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0929 19:04:34.661377 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:04:34.661586 (+ 209us) server_negotiation.cc:407] Connection header received
0929 19:04:34.662465 (+ 879us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:04:34.662466 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:04:34.662476 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:04:34.662526 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:04:34.663038 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:34.663525 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:34.664401 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:34.664573 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:34.664619 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:04:34.666487 (+ 1868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:04:34.666489 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:04:34.666489 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:04:34.666495 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:04:34.667885 (+ 1390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:04:34.668334 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:04:34.668335 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:04:34.668335 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:04:34.668385 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:04:34.668639 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:04:34.668639 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:04:34.668640 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:04:34.668725 (+ 85us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:04:34.668769 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0929 19:04:34.669060 (+ 291us) server_negotiation.cc:299] Negotiation successful
0929 19:04:34.669119 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260929 19:04:34.669451 18805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:34.661302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45477 (local address 127.0.98.2:33463)
0929 19:04:34.661474 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0929 19:04:34.661506 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0929 19:04:34.662374 (+ 868us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:04:34.662572 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:04:34.662575 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:04:34.662795 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:04:34.662973 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:04:34.662977 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:04:34.663594 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:04:34.663594 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:04:34.664325 (+ 731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:04:34.664327 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:04:34.664636 (+ 309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:04:34.664637 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:04:34.664725 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:04:34.665151 (+ 426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:04:34.665158 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:04:34.666408 (+ 1250us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:04:34.667958 (+ 1550us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:04:34.667964 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:04:34.667965 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:04:34.668259 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:04:34.668459 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:04:34.668460 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:04:34.668461 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:04:34.668580 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:04:34.668782 (+ 202us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:04:34.668787 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:04:34.668942 (+ 155us) client_negotiation.cc:769] Sending connection context
0929 19:04:34.669042 (+ 100us) client_negotiation.cc:240] Negotiation successful
0929 19:04:34.669189 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":82,"threads_started":1}
I20260929 19:04:34.669903 18803 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45477
I20260929 19:04:34.669998 18803 heartbeater.cc:464] Registering TS with master...
I20260929 19:04:34.670166 18803 heartbeater.cc:511] Master 127.0.98.62:45477 requested a full tablet report, sending...
I20260929 19:04:34.670511 18478 ts_manager.cc:194] Registered new tserver with Master: f701558d57854f36b7bc20434382737c (127.0.98.2:37553)
I20260929 19:04:34.671005 18478 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:33463
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708447806542-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708447806542-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260929 19:04:34.689437 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUbnLHE/build/release/bin/kudu
/tmp/dist-test-taskUbnLHE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708447806542-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708447806542-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708447806542-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708447806542-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45477
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708447806542-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708447806542-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708447806542-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708447806542-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:04:34.763826 18808 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:04:34.763983 18808 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:04:34.764003 18808 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:04:34.765081 18808 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:04:34.765115 18808 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:04:34.765139 18808 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 19:04:34.766494 18808 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708447806542-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708447806542-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708447806542-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708447806542-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45477
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18808
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708447806542-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 18:09:13 UTC on ccycloud.quasar-hpkalf.root.comops.site
build id 4680913
I20260929 19:04:34.766708 18808 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:04:34.766929 18808 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:04:34.771502 18816 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:04:34.771528 18808 server_base.cc:1034] running on GCE node
W20260929 19:04:34.771678 18813 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:04:34.771625 18814 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:04:34.771888 18808 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:04:34.772127 18808 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:04:34.773265 18808 hybrid_clock.cc:648] HybridClock initialized: now 1790708674773231 us; error 44 us; skew 500 ppm
Sep 29 19:04:34 dist-test-slave-lqkv krb5kdc[18433](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708674, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:04:34.775413 18808 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 19:04:34.775763 18808 webserver.cc:460] Webserver started at http://127.0.98.3:46651/ using document root <none> and password file <none>
I20260929 19:04:34.775964 18808 fs_manager.cc:359] Metadata directory not provided
I20260929 19:04:34.776023 18808 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:04:34.776158 18808 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:04:34.777173 18808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708447806542-392-0/minicluster-data/ts-2/data/instance:
uuid: "9a8e48ef54664a4192aac0419015e227"
format_stamp: "Formatted at 2026-09-29 19:04:34 on dist-test-slave-lqkv"
server_key: "9b84238becd6460e5504cf76d4eccecd"
server_key_iv: "86ba6544db932957d420ad0c5c6e944c"
server_key_version: "encryptionkey@0"
I20260929 19:04:34.777523 18808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708447806542-392-0/minicluster-data/ts-2/wal/instance:
uuid: "9a8e48ef54664a4192aac0419015e227"
format_stamp: "Formatted at 2026-09-29 19:04:34 on dist-test-slave-lqkv"
server_key: "9b84238becd6460e5504cf76d4eccecd"
server_key_iv: "86ba6544db932957d420ad0c5c6e944c"
server_key_version: "encryptionkey@0"
I20260929 19:04:34.778630 18808 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 19:04:34.779227 18823 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:04:34.779371 18808 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 19:04:34.779455 18808 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708447806542-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708447806542-392-0/minicluster-data/ts-2/wal
uuid: "9a8e48ef54664a4192aac0419015e227"
format_stamp: "Formatted at 2026-09-29 19:04:34 on dist-test-slave-lqkv"
server_key: "9b84238becd6460e5504cf76d4eccecd"
server_key_iv: "86ba6544db932957d420ad0c5c6e944c"
server_key_version: "encryptionkey@0"
I20260929 19:04:34.779529 18808 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708447806542-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708447806542-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708447806542-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:04:34.798374 18808 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:04:34.799832 18808 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:04:34.799921 18808 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:04:34.800057 18808 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:04:34.800246 18808 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:04:34.800271 18808 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:04:34.800288 18808 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:04:34.800299 18808 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:04:34.805284 18808 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37463
I20260929 19:04:34.805361 18936 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37463 every 8 connection(s)
I20260929 19:04:34.805632 18808 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708447806542-392-0/minicluster-data/ts-2/data/info.pb
Sep 29 19:04:34 dist-test-slave-lqkv krb5kdc[18433](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708674, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 19:04:34.813273 18518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:34.806312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52889 (local address 127.0.98.62:45477)
0929 19:04:34.806398 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0929 19:04:34.806401 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 19:04:34.806509 (+ 108us) server_negotiation.cc:407] Connection header received
0929 19:04:34.807310 (+ 801us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:04:34.807311 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:04:34.807321 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:04:34.807369 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:04:34.807880 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:34.808374 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:34.809277 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:34.809435 (+ 158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:34.809485 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:04:34.811170 (+ 1685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:04:34.811171 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:04:34.811171 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:04:34.811176 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:04:34.812252 (+ 1076us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:04:34.812548 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:04:34.812549 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:04:34.812549 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:04:34.812592 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:04:34.812776 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:04:34.812777 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:04:34.812777 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:04:34.812913 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:04:34.812960 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0929 19:04:34.813148 (+ 188us) server_negotiation.cc:299] Negotiation successful
0929 19:04:34.813199 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 19:04:34.813370 18939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:34.806248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45477 (local address 127.0.98.3:52889)
0929 19:04:34.806384 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0929 19:04:34.806428 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 19:04:34.807217 (+ 789us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:04:34.807432 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:04:34.807435 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:04:34.807647 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:04:34.807812 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:04:34.807815 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:04:34.808455 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:04:34.808455 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:04:34.809202 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:04:34.809205 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:04:34.809500 (+ 295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:04:34.809501 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:04:34.809578 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:04:34.809989 (+ 411us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:04:34.810002 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:04:34.811099 (+ 1097us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:04:34.812315 (+ 1216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:04:34.812320 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:04:34.812321 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:04:34.812491 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:04:34.812640 (+ 149us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:04:34.812640 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:04:34.812641 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:04:34.812716 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:04:34.812963 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:04:34.812967 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:04:34.813053 (+ 86us) client_negotiation.cc:769] Sending connection context
0929 19:04:34.813121 (+ 68us) client_negotiation.cc:240] Negotiation successful
0929 19:04:34.813212 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":66,"threads_started":1}
I20260929 19:04:34.813455 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUbnLHE/build/release/bin/kudu as pid 18808
I20260929 19:04:34.813556 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708447806542-392-0/minicluster-data/ts-2/wal/instance
I20260929 19:04:34.813685 392 external_mini_cluster.cc:1428] Setting key b1ae09a1c6fc6c247f2ee55cfec6e4e7
I20260929 19:04:34.813953 18937 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45477
I20260929 19:04:34.814038 18937 heartbeater.cc:464] Registering TS with master...
I20260929 19:04:34.814198 18937 heartbeater.cc:511] Master 127.0.98.62:45477 requested a full tablet report, sending...
I20260929 19:04:34.814584 18478 ts_manager.cc:194] Registered new tserver with Master: 9a8e48ef54664a4192aac0419015e227 (127.0.98.3:37463)
I20260929 19:04:34.814929 18478 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:52889
I20260929 19:04:34.825420 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 19:04:34.831703 18518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:34.826656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33688 (local address 127.0.98.62:45477)
0929 19:04:34.826707 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0929 19:04:34.826709 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:04:34.826789 (+ 80us) server_negotiation.cc:407] Connection header received
0929 19:04:34.826815 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:04:34.826815 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:04:34.826823 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:04:34.826879 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:04:34.827388 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:34.827820 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:34.828502 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:34.828652 (+ 150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:34.828694 (+ 42us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:04:34.829857 (+ 1163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:04:34.829858 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:04:34.829858 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:04:34.829863 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:04:34.830844 (+ 981us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:04:34.831159 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:04:34.831159 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:04:34.831160 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:04:34.831201 (+ 41us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:04:34.831363 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:04:34.831364 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:04:34.831364 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:04:34.831450 (+ 86us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:04:34.831513 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0929 19:04:34.831613 (+ 100us) server_negotiation.cc:299] Negotiation successful
0929 19:04:34.831653 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260929 19:04:34.832911 18478 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33688:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 19:04:34.834239 18478 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 19:04:34.839598 18947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:34.835855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43765 (local address 127.0.0.1:34468)
0929 19:04:34.835992 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0929 19:04:34.836000 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0929 19:04:34.836034 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:04:34.836658 (+ 624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:04:34.836658 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:04:34.836660 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:04:34.836712 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:04:34.836714 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:04:34.838077 (+ 1363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:04:34.838078 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:04:34.838767 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:04:34.838770 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:04:34.839361 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:04:34.839362 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:04:34.839427 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:04:34.839438 (+ 11us) client_negotiation.cc:769] Sending connection context
0929 19:04:34.839498 (+ 60us) client_negotiation.cc:240] Negotiation successful
0929 19:04:34.839541 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":57,"threads_started":1}
I20260929 19:04:34.839607 18950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:34.836272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34468 (local address 127.0.98.1:43765)
0929 19:04:34.836459 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0929 19:04:34.836462 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 19:04:34.836473 (+ 11us) server_negotiation.cc:407] Connection header received
0929 19:04:34.836548 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:04:34.836551 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:04:34.836568 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:04:34.836648 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:04:34.837085 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:34.837986 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:34.838841 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:34.839284 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:34.839360 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:04:34.839417 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0929 19:04:34.839511 (+ 94us) server_negotiation.cc:299] Negotiation successful
0929 19:04:34.839545 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":85,"threads_started":1}
I20260929 19:04:34.839833 18945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:34.835847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37553 (local address 127.0.0.1:55264)
0929 19:04:34.835943 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0929 19:04:34.835973 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0929 19:04:34.836030 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:04:34.836420 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:04:34.836423 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:04:34.836425 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:04:34.836494 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:04:34.836496 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:04:34.837790 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:04:34.837791 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:04:34.838791 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:04:34.838793 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:04:34.839605 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:04:34.839606 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:04:34.839661 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:04:34.839686 (+ 25us) client_negotiation.cc:769] Sending connection context
0929 19:04:34.839730 (+ 44us) client_negotiation.cc:240] Negotiation successful
0929 19:04:34.839766 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":78,"mutex_wait_us":3,"thread_start_us":60,"threads_started":1}
I20260929 19:04:34.839872 18949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:34.836082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55264 (local address 127.0.98.2:37553)
0929 19:04:34.836226 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0929 19:04:34.836231 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 19:04:34.836242 (+ 11us) server_negotiation.cc:407] Connection header received
0929 19:04:34.836281 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:04:34.836283 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:04:34.836298 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:04:34.836399 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:04:34.836584 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:34.837691 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:34.838882 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:34.839476 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:34.839534 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:04:34.839575 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0929 19:04:34.839752 (+ 177us) server_negotiation.cc:299] Negotiation successful
0929 19:04:34.839797 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":66,"threads_started":1}
I20260929 19:04:34.840041 18948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:34.836037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44322 (local address 127.0.98.3:37463)
0929 19:04:34.836168 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0929 19:04:34.836171 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 19:04:34.836514 (+ 343us) server_negotiation.cc:407] Connection header received
0929 19:04:34.836579 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:04:34.836581 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:04:34.836594 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:04:34.836690 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:04:34.836904 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:34.837556 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:34.838880 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:34.839476 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:34.839518 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:04:34.839562 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0929 19:04:34.839896 (+ 334us) server_negotiation.cc:299] Negotiation successful
0929 19:04:34.839958 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":73,"threads_started":1}
I20260929 19:04:34.840243 18946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:34.835858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37463 (local address 127.0.0.1:44322)
0929 19:04:34.836455 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0929 19:04:34.836466 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 19:04:34.836519 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:04:34.836768 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:04:34.836769 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:04:34.836771 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:04:34.836798 (+ 27us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:04:34.836799 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:04:34.837645 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:04:34.837645 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:04:34.838792 (+ 1147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:04:34.838793 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:04:34.839610 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:04:34.839611 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:04:34.839661 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:04:34.839815 (+ 154us) client_negotiation.cc:769] Sending connection context
0929 19:04:34.840136 (+ 321us) client_negotiation.cc:240] Negotiation successful
0929 19:04:34.840181 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":584,"mutex_wait_us":174,"thread_start_us":66,"threads_started":1}
I20260929 19:04:34.840407 18603 tablet_service.cc:1467] Processing CreateTablet for tablet a6129f0697eb428f8472823f7c726737 (DEFAULT_TABLE table=test-table [id=2b60b6019dab49ba97d1fe27f6cc1ccc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:04:34.840718 18603 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6129f0697eb428f8472823f7c726737. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:04:34.840760 18871 tablet_service.cc:1467] Processing CreateTablet for tablet a6129f0697eb428f8472823f7c726737 (DEFAULT_TABLE table=test-table [id=2b60b6019dab49ba97d1fe27f6cc1ccc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:04:34.840781 18737 tablet_service.cc:1467] Processing CreateTablet for tablet a6129f0697eb428f8472823f7c726737 (DEFAULT_TABLE table=test-table [id=2b60b6019dab49ba97d1fe27f6cc1ccc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:04:34.841060 18871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6129f0697eb428f8472823f7c726737. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:04:34.841091 18737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6129f0697eb428f8472823f7c726737. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:04:34.843408 18951 tablet_bootstrap.cc:492] T a6129f0697eb428f8472823f7c726737 P 9a8e48ef54664a4192aac0419015e227: Bootstrap starting.
I20260929 19:04:34.844034 18951 tablet_bootstrap.cc:654] T a6129f0697eb428f8472823f7c726737 P 9a8e48ef54664a4192aac0419015e227: Neither blocks nor log segments found. Creating new log.
I20260929 19:04:34.844070 18953 tablet_bootstrap.cc:492] T a6129f0697eb428f8472823f7c726737 P f701558d57854f36b7bc20434382737c: Bootstrap starting.
I20260929 19:04:34.844354 18952 tablet_bootstrap.cc:492] T a6129f0697eb428f8472823f7c726737 P 2cd8d48de66142be8a36e8278b01ed16: Bootstrap starting.
I20260929 19:04:34.844630 18951 log.cc:824] T a6129f0697eb428f8472823f7c726737 P 9a8e48ef54664a4192aac0419015e227: Log is configured to *not* fsync() on all Append() calls
I20260929 19:04:34.844736 18953 tablet_bootstrap.cc:654] T a6129f0697eb428f8472823f7c726737 P f701558d57854f36b7bc20434382737c: Neither blocks nor log segments found. Creating new log.
I20260929 19:04:34.844990 18952 tablet_bootstrap.cc:654] T a6129f0697eb428f8472823f7c726737 P 2cd8d48de66142be8a36e8278b01ed16: Neither blocks nor log segments found. Creating new log.
I20260929 19:04:34.845274 18953 log.cc:824] T a6129f0697eb428f8472823f7c726737 P f701558d57854f36b7bc20434382737c: Log is configured to *not* fsync() on all Append() calls
I20260929 19:04:34.845517 18952 log.cc:824] T a6129f0697eb428f8472823f7c726737 P 2cd8d48de66142be8a36e8278b01ed16: Log is configured to *not* fsync() on all Append() calls
I20260929 19:04:34.846000 18953 tablet_bootstrap.cc:492] T a6129f0697eb428f8472823f7c726737 P f701558d57854f36b7bc20434382737c: No bootstrap required, opened a new log
I20260929 19:04:34.846071 18953 ts_tablet_manager.cc:1397] T a6129f0697eb428f8472823f7c726737 P f701558d57854f36b7bc20434382737c: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260929 19:04:34.846135 18951 tablet_bootstrap.cc:492] T a6129f0697eb428f8472823f7c726737 P 9a8e48ef54664a4192aac0419015e227: No bootstrap required, opened a new log
I20260929 19:04:34.846210 18951 ts_tablet_manager.cc:1397] T a6129f0697eb428f8472823f7c726737 P 9a8e48ef54664a4192aac0419015e227: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260929 19:04:34.846300 18952 tablet_bootstrap.cc:492] T a6129f0697eb428f8472823f7c726737 P 2cd8d48de66142be8a36e8278b01ed16: No bootstrap required, opened a new log
I20260929 19:04:34.846366 18952 ts_tablet_manager.cc:1397] T a6129f0697eb428f8472823f7c726737 P 2cd8d48de66142be8a36e8278b01ed16: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 19:04:34.847757 18953 raft_consensus.cc:348] T a6129f0697eb428f8472823f7c726737 P f701558d57854f36b7bc20434382737c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cd8d48de66142be8a36e8278b01ed16" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43765 } } peers { permanent_uuid: "f701558d57854f36b7bc20434382737c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37553 } } peers { permanent_uuid: "9a8e48ef54664a4192aac0419015e227" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37463 } }
I20260929 19:04:34.847913 18953 raft_consensus.cc:374] T a6129f0697eb428f8472823f7c726737 P f701558d57854f36b7bc20434382737c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:04:34.847957 18953 raft_consensus.cc:729] T a6129f0697eb428f8472823f7c726737 P f701558d57854f36b7bc20434382737c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f701558d57854f36b7bc20434382737c, State: Initialized, Role: FOLLOWER
I20260929 19:04:34.848063 18953 consensus_queue.cc:260] T a6129f0697eb428f8472823f7c726737 P f701558d57854f36b7bc20434382737c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cd8d48de66142be8a36e8278b01ed16" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43765 } } peers { permanent_uuid: "f701558d57854f36b7bc20434382737c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37553 } } peers { permanent_uuid: "9a8e48ef54664a4192aac0419015e227" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37463 } }
I20260929 19:04:34.848160 18951 raft_consensus.cc:348] T a6129f0697eb428f8472823f7c726737 P 9a8e48ef54664a4192aac0419015e227 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cd8d48de66142be8a36e8278b01ed16" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43765 } } peers { permanent_uuid: "f701558d57854f36b7bc20434382737c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37553 } } peers { permanent_uuid: "9a8e48ef54664a4192aac0419015e227" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37463 } }
I20260929 19:04:34.848230 18952 raft_consensus.cc:348] T a6129f0697eb428f8472823f7c726737 P 2cd8d48de66142be8a36e8278b01ed16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cd8d48de66142be8a36e8278b01ed16" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43765 } } peers { permanent_uuid: "f701558d57854f36b7bc20434382737c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37553 } } peers { permanent_uuid: "9a8e48ef54664a4192aac0419015e227" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37463 } }
I20260929 19:04:34.848330 18951 raft_consensus.cc:374] T a6129f0697eb428f8472823f7c726737 P 9a8e48ef54664a4192aac0419015e227 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:04:34.848348 18953 ts_tablet_manager.cc:1428] T a6129f0697eb428f8472823f7c726737 P f701558d57854f36b7bc20434382737c: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260929 19:04:34.848377 18951 raft_consensus.cc:729] T a6129f0697eb428f8472823f7c726737 P 9a8e48ef54664a4192aac0419015e227 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a8e48ef54664a4192aac0419015e227, State: Initialized, Role: FOLLOWER
I20260929 19:04:34.848430 18952 raft_consensus.cc:374] T a6129f0697eb428f8472823f7c726737 P 2cd8d48de66142be8a36e8278b01ed16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:04:34.848466 18952 raft_consensus.cc:729] T a6129f0697eb428f8472823f7c726737 P 2cd8d48de66142be8a36e8278b01ed16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2cd8d48de66142be8a36e8278b01ed16, State: Initialized, Role: FOLLOWER
I20260929 19:04:34.848479 18803 heartbeater.cc:503] Master 127.0.98.62:45477 was elected leader, sending a full tablet report...
I20260929 19:04:34.848557 18952 consensus_queue.cc:260] T a6129f0697eb428f8472823f7c726737 P 2cd8d48de66142be8a36e8278b01ed16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cd8d48de66142be8a36e8278b01ed16" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43765 } } peers { permanent_uuid: "f701558d57854f36b7bc20434382737c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37553 } } peers { permanent_uuid: "9a8e48ef54664a4192aac0419015e227" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37463 } }
I20260929 19:04:34.848832 18952 ts_tablet_manager.cc:1428] T a6129f0697eb428f8472823f7c726737 P 2cd8d48de66142be8a36e8278b01ed16: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260929 19:04:34.848839 18951 consensus_queue.cc:260] T a6129f0697eb428f8472823f7c726737 P 9a8e48ef54664a4192aac0419015e227 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cd8d48de66142be8a36e8278b01ed16" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43765 } } peers { permanent_uuid: "f701558d57854f36b7bc20434382737c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37553 } } peers { permanent_uuid: "9a8e48ef54664a4192aac0419015e227" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37463 } }
I20260929 19:04:34.848973 18669 heartbeater.cc:503] Master 127.0.98.62:45477 was elected leader, sending a full tablet report...
I20260929 19:04:34.849085 18951 ts_tablet_manager.cc:1428] T a6129f0697eb428f8472823f7c726737 P 9a8e48ef54664a4192aac0419015e227: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260929 19:04:34.849167 18937 heartbeater.cc:503] Master 127.0.98.62:45477 was elected leader, sending a full tablet report...
W20260929 19:04:34.911211 18804 tablet.cc:2369] T a6129f0697eb428f8472823f7c726737 P f701558d57854f36b7bc20434382737c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 19:04:35.004902 18670 tablet.cc:2369] T a6129f0697eb428f8472823f7c726737 P 2cd8d48de66142be8a36e8278b01ed16: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 19:04:35.056140 18938 tablet.cc:2369] T a6129f0697eb428f8472823f7c726737 P 9a8e48ef54664a4192aac0419015e227: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 19:04:35.166625 18958 raft_consensus.cc:482] T a6129f0697eb428f8472823f7c726737 P 2cd8d48de66142be8a36e8278b01ed16 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 19:04:35.166746 18958 raft_consensus.cc:504] T a6129f0697eb428f8472823f7c726737 P 2cd8d48de66142be8a36e8278b01ed16 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cd8d48de66142be8a36e8278b01ed16" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43765 } } peers { permanent_uuid: "f701558d57854f36b7bc20434382737c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37553 } } peers { permanent_uuid: "9a8e48ef54664a4192aac0419015e227" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37463 } }
I20260929 19:04:35.167169 18958 leader_election.cc:288] T a6129f0697eb428f8472823f7c726737 P 2cd8d48de66142be8a36e8278b01ed16 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f701558d57854f36b7bc20434382737c (127.0.98.2:37553), 9a8e48ef54664a4192aac0419015e227 (127.0.98.3:37463)
I20260929 19:04:35.169947 18948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:35.167422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41345 (local address 127.0.98.3:37463)
0929 19:04:35.167466 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0929 19:04:35.167468 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:04:35.167569 (+ 101us) server_negotiation.cc:407] Connection header received
0929 19:04:35.167651 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:04:35.167652 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:04:35.167660 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:04:35.167724 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:04:35.167909 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:35.168403 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:35.169297 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:35.169642 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:35.169717 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:04:35.169727 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0929 19:04:35.169849 (+ 122us) server_negotiation.cc:299] Negotiation successful
0929 19:04:35.169891 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260929 19:04:35.170013 18961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:35.167331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37463 (local address 127.0.98.1:41345)
0929 19:04:35.167486 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0929 19:04:35.167497 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 19:04:35.167582 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:04:35.167741 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:04:35.167741 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:04:35.167743 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:04:35.167831 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:04:35.167833 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:04:35.168496 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:04:35.168497 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:04:35.169220 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:04:35.169222 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:04:35.169717 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:04:35.169718 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:04:35.169785 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:04:35.169788 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 19:04:35.169844 (+ 56us) client_negotiation.cc:240] Negotiation successful
0929 19:04:35.169884 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":64,"threads_started":1}
I20260929 19:04:35.170029 18960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:35.167357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37553 (local address 127.0.98.1:40737)
0929 19:04:35.167485 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0929 19:04:35.167495 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 19:04:35.167585 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:04:35.167730 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:04:35.167731 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:04:35.167733 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:04:35.167802 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:04:35.167806 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:04:35.168472 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:04:35.168473 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:04:35.169220 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:04:35.169222 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:04:35.169764 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:04:35.169765 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:04:35.169826 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:04:35.169829 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 19:04:35.169895 (+ 66us) client_negotiation.cc:240] Negotiation successful
0929 19:04:35.169929 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"mutex_wait_us":10,"thread_start_us":53,"threads_started":1}
I20260929 19:04:35.170011 18949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:35.167424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40737 (local address 127.0.98.2:37553)
0929 19:04:35.167464 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0929 19:04:35.167465 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 19:04:35.167566 (+ 101us) server_negotiation.cc:407] Connection header received
0929 19:04:35.167640 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:04:35.167641 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:04:35.167653 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:04:35.167726 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:04:35.167906 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:35.168388 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:35.169319 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:35.169664 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:35.169726 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:04:35.169735 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0929 19:04:35.169904 (+ 169us) server_negotiation.cc:299] Negotiation successful
0929 19:04:35.169934 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260929 19:04:35.170279 18757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6129f0697eb428f8472823f7c726737" candidate_uuid: "2cd8d48de66142be8a36e8278b01ed16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f701558d57854f36b7bc20434382737c" is_pre_election: true
I20260929 19:04:35.170428 18757 raft_consensus.cc:2454] T a6129f0697eb428f8472823f7c726737 P f701558d57854f36b7bc20434382737c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2cd8d48de66142be8a36e8278b01ed16 in term 0.
I20260929 19:04:35.170413 18891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6129f0697eb428f8472823f7c726737" candidate_uuid: "2cd8d48de66142be8a36e8278b01ed16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a8e48ef54664a4192aac0419015e227" is_pre_election: true
I20260929 19:04:35.170576 18891 raft_consensus.cc:2454] T a6129f0697eb428f8472823f7c726737 P 9a8e48ef54664a4192aac0419015e227 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2cd8d48de66142be8a36e8278b01ed16 in term 0.
I20260929 19:04:35.170612 18558 leader_election.cc:302] T a6129f0697eb428f8472823f7c726737 P 2cd8d48de66142be8a36e8278b01ed16 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2cd8d48de66142be8a36e8278b01ed16, f701558d57854f36b7bc20434382737c; no voters:
I20260929 19:04:35.170737 18958 raft_consensus.cc:2781] T a6129f0697eb428f8472823f7c726737 P 2cd8d48de66142be8a36e8278b01ed16 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 19:04:35.170795 18958 raft_consensus.cc:482] T a6129f0697eb428f8472823f7c726737 P 2cd8d48de66142be8a36e8278b01ed16 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 19:04:35.170811 18958 raft_consensus.cc:3037] T a6129f0697eb428f8472823f7c726737 P 2cd8d48de66142be8a36e8278b01ed16 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:04:35.171468 18958 raft_consensus.cc:504] T a6129f0697eb428f8472823f7c726737 P 2cd8d48de66142be8a36e8278b01ed16 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cd8d48de66142be8a36e8278b01ed16" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43765 } } peers { permanent_uuid: "f701558d57854f36b7bc20434382737c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37553 } } peers { permanent_uuid: "9a8e48ef54664a4192aac0419015e227" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37463 } }
I20260929 19:04:35.171583 18958 leader_election.cc:288] T a6129f0697eb428f8472823f7c726737 P 2cd8d48de66142be8a36e8278b01ed16 [CANDIDATE]: Term 1 election: Requested vote from peers f701558d57854f36b7bc20434382737c (127.0.98.2:37553), 9a8e48ef54664a4192aac0419015e227 (127.0.98.3:37463)
I20260929 19:04:35.171707 18757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6129f0697eb428f8472823f7c726737" candidate_uuid: "2cd8d48de66142be8a36e8278b01ed16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f701558d57854f36b7bc20434382737c"
I20260929 19:04:35.171721 18891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6129f0697eb428f8472823f7c726737" candidate_uuid: "2cd8d48de66142be8a36e8278b01ed16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a8e48ef54664a4192aac0419015e227"
I20260929 19:04:35.171790 18891 raft_consensus.cc:3037] T a6129f0697eb428f8472823f7c726737 P 9a8e48ef54664a4192aac0419015e227 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:04:35.171792 18757 raft_consensus.cc:3037] T a6129f0697eb428f8472823f7c726737 P f701558d57854f36b7bc20434382737c [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:04:35.172384 18891 raft_consensus.cc:2454] T a6129f0697eb428f8472823f7c726737 P 9a8e48ef54664a4192aac0419015e227 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2cd8d48de66142be8a36e8278b01ed16 in term 1.
I20260929 19:04:35.172394 18757 raft_consensus.cc:2454] T a6129f0697eb428f8472823f7c726737 P f701558d57854f36b7bc20434382737c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2cd8d48de66142be8a36e8278b01ed16 in term 1.
I20260929 19:04:35.172538 18556 leader_election.cc:302] T a6129f0697eb428f8472823f7c726737 P 2cd8d48de66142be8a36e8278b01ed16 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2cd8d48de66142be8a36e8278b01ed16, 9a8e48ef54664a4192aac0419015e227; no voters:
I20260929 19:04:35.172631 18958 raft_consensus.cc:2781] T a6129f0697eb428f8472823f7c726737 P 2cd8d48de66142be8a36e8278b01ed16 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:04:35.172755 18958 raft_consensus.cc:686] T a6129f0697eb428f8472823f7c726737 P 2cd8d48de66142be8a36e8278b01ed16 [term 1 LEADER]: Becoming Leader. State: Replica: 2cd8d48de66142be8a36e8278b01ed16, State: Running, Role: LEADER
I20260929 19:04:35.172856 18958 consensus_queue.cc:237] T a6129f0697eb428f8472823f7c726737 P 2cd8d48de66142be8a36e8278b01ed16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cd8d48de66142be8a36e8278b01ed16" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43765 } } peers { permanent_uuid: "f701558d57854f36b7bc20434382737c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37553 } } peers { permanent_uuid: "9a8e48ef54664a4192aac0419015e227" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37463 } }
I20260929 19:04:35.173630 18477 catalog_manager.cc:5495] T a6129f0697eb428f8472823f7c726737 P 2cd8d48de66142be8a36e8278b01ed16 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2cd8d48de66142be8a36e8278b01ed16 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "2cd8d48de66142be8a36e8278b01ed16" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cd8d48de66142be8a36e8278b01ed16" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43765 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f701558d57854f36b7bc20434382737c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37553 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a8e48ef54664a4192aac0419015e227" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37463 } health_report { overall_health: UNKNOWN } } }
I20260929 19:04:35.192682 18950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:35.190694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34480 (local address 127.0.98.1:43765)
0929 19:04:35.190744 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0929 19:04:35.190745 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 19:04:35.190752 (+ 7us) server_negotiation.cc:407] Connection header received
0929 19:04:35.190822 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:04:35.190823 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:04:35.190836 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:04:35.190895 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 19:04:35.191008 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:35.191414 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:35.192059 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:35.192208 (+ 149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:35.192249 (+ 41us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:04:35.192369 (+ 120us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 19:04:35.192438 (+ 69us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 19:04:35.192495 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0929 19:04:35.192567 (+ 72us) server_negotiation.cc:299] Negotiation successful
0929 19:04:35.192575 (+ 8us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260929 19:04:35.194121 18757 raft_consensus.cc:1261] T a6129f0697eb428f8472823f7c726737 P f701558d57854f36b7bc20434382737c [term 1 FOLLOWER]: Refusing update from remote peer 2cd8d48de66142be8a36e8278b01ed16: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 19:04:35.194125 18891 raft_consensus.cc:1261] T a6129f0697eb428f8472823f7c726737 P 9a8e48ef54664a4192aac0419015e227 [term 1 FOLLOWER]: Refusing update from remote peer 2cd8d48de66142be8a36e8278b01ed16: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 19:04:35.194447 18958 consensus_queue.cc:1035] T a6129f0697eb428f8472823f7c726737 P 2cd8d48de66142be8a36e8278b01ed16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f701558d57854f36b7bc20434382737c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 19:04:35.194593 18962 consensus_queue.cc:1035] T a6129f0697eb428f8472823f7c726737 P 2cd8d48de66142be8a36e8278b01ed16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a8e48ef54664a4192aac0419015e227" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 19:04:35.196991 18966 mvcc.cc:205] Tried to move back new op lower bound from 7334742733593079808 to 7334742733508677632. Current Snapshot: MvccSnapshot[applied={T|T < 7334742733593079808}]
I20260929 19:04:35.197249 18967 mvcc.cc:205] Tried to move back new op lower bound from 7334742733593079808 to 7334742733508677632. Current Snapshot: MvccSnapshot[applied={T|T < 7334742733593079808}]
I20260929 19:04:35.198572 18478 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33688:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 19:04:35.198659 18478 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33688:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 19:04:35.199330 18478 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9325793030b244efb80f847990e12a95: Sending DeleteTablet for 3 replicas of tablet a6129f0697eb428f8472823f7c726737
I20260929 19:04:35.199656 18871 tablet_service.cc:1514] Processing DeleteTablet for tablet a6129f0697eb428f8472823f7c726737 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:04:35 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44322
I20260929 19:04:35.199664 18737 tablet_service.cc:1514] Processing DeleteTablet for tablet a6129f0697eb428f8472823f7c726737 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:04:35 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55264
I20260929 19:04:35.199661 18603 tablet_service.cc:1514] Processing DeleteTablet for tablet a6129f0697eb428f8472823f7c726737 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:04:35 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34468
I20260929 19:04:35.199841 18975 tablet_replica.cc:330] T a6129f0697eb428f8472823f7c726737 P 9a8e48ef54664a4192aac0419015e227: stopping tablet replica
I20260929 19:04:35.199867 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUbnLHE/build/release/bin/kudu with pid 18540
I20260929 19:04:35.200414 18975 raft_consensus.cc:2229] T a6129f0697eb428f8472823f7c726737 P 9a8e48ef54664a4192aac0419015e227 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 19:04:35.200519 18975 raft_consensus.cc:2258] T a6129f0697eb428f8472823f7c726737 P 9a8e48ef54664a4192aac0419015e227 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 19:04:35.200677 18976 tablet_replica.cc:330] T a6129f0697eb428f8472823f7c726737 P f701558d57854f36b7bc20434382737c: stopping tablet replica
I20260929 19:04:35.200755 18976 raft_consensus.cc:2229] T a6129f0697eb428f8472823f7c726737 P f701558d57854f36b7bc20434382737c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 19:04:35.200874 18976 raft_consensus.cc:2258] T a6129f0697eb428f8472823f7c726737 P f701558d57854f36b7bc20434382737c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 19:04:35.201220 18975 ts_tablet_manager.cc:1905] T a6129f0697eb428f8472823f7c726737 P 9a8e48ef54664a4192aac0419015e227: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 19:04:35.202888 18975 ts_tablet_manager.cc:1918] T a6129f0697eb428f8472823f7c726737 P 9a8e48ef54664a4192aac0419015e227: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 19:04:35.202957 18975 log.cc:1196] T a6129f0697eb428f8472823f7c726737 P 9a8e48ef54664a4192aac0419015e227: Deleting WAL directory at /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790708447806542-392-0/minicluster-data/ts-2/wal/wals/a6129f0697eb428f8472823f7c726737
I20260929 19:04:35.203186 18975 ts_tablet_manager.cc:1939] T a6129f0697eb428f8472823f7c726737 P 9a8e48ef54664a4192aac0419015e227: Deleting consensus metadata
I20260929 19:04:35.203537 18462 catalog_manager.cc:4841] TS 9a8e48ef54664a4192aac0419015e227 (127.0.98.3:37463): tablet a6129f0697eb428f8472823f7c726737 (table test-table [id=2b60b6019dab49ba97d1fe27f6cc1ccc]) successfully deleted
I20260929 19:04:35.205082 18976 ts_tablet_manager.cc:1905] T a6129f0697eb428f8472823f7c726737 P f701558d57854f36b7bc20434382737c: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260929 19:04:35.205475 18465 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:43765 (error 108)
I20260929 19:04:35.205925 18946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:35.205786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43765 (local address 127.0.0.1:34490)
0929 19:04:35.205853 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0929 19:04:35.205892 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260929 19:04:35.205979 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUbnLHE/build/release/bin/kudu with pid 18674
W20260929 19:04:35.206598 18465 catalog_manager.cc:4584] TS 2cd8d48de66142be8a36e8278b01ed16 (127.0.98.1:43765): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a6129f0697eb428f8472823f7c726737: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43765: connect: Connection refused (error 111)
I20260929 19:04:35.211040 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUbnLHE/build/release/bin/kudu with pid 18808
I20260929 19:04:35.211073 18946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:35.210980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37553 (local address 127.0.0.1:55274)
0929 19:04:35.211028 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0929 19:04:35.211054 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:37553: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260929 19:04:35.215097 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUbnLHE/build/release/bin/kudu with pid 18446
2026-09-29T19:04:35Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2049 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-29T19:04:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T19:04:35Z Disabled control of system clock
I20260929 19:04:35.240008 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUbnLHE/build/release/bin/kudu
/tmp/dist-test-taskUbnLHE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708447806542-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708447806542-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708447806542-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708447806542-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37853
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:38625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37853
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 19:04:35.312948 18981 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:04:35.313195 18981 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:04:35.313228 18981 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:04:35.314219 18981 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 19:04:35.314250 18981 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:04:35.314261 18981 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:04:35.314273 18981 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 19:04:35.314283 18981 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 19:04:35.315431 18981 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708447806542-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708447806542-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37853
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37853
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708447806542-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18981
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708447806542-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 18:09:13 UTC on ccycloud.quasar-hpkalf.root.comops.site
build id 4680913
I20260929 19:04:35.315578 18981 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:04:35.315753 18981 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:04:35.320169 18989 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:04:35.320194 18987 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:04:35.320228 18986 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:04:35.320467 18981 server_base.cc:1034] running on GCE node
I20260929 19:04:35.320693 18981 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:04:35.321002 18981 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:04:35.322162 18981 hybrid_clock.cc:648] HybridClock initialized: now 1790708675322152 us; error 38 us; skew 500 ppm
I20260929 19:04:35.323391 18981 webserver.cc:460] Webserver started at http://127.0.98.62:46725/ using document root <none> and password file <none>
I20260929 19:04:35.323609 18981 fs_manager.cc:359] Metadata directory not provided
I20260929 19:04:35.323673 18981 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:04:35.323796 18981 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:04:35.324790 18981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708447806542-392-0/minicluster-data/master-0/data/instance:
uuid: "1524c320cdc84d5a886b598280a7a7ab"
format_stamp: "Formatted at 2026-09-29 19:04:35 on dist-test-slave-lqkv"
server_key: "1ae97fa41683907917110d13ad171669"
server_key_iv: "b3e6a62e33fb0c8ac27bd29db63477b1"
server_key_version: "encryptionkey@0"
I20260929 19:04:35.325207 18981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708447806542-392-0/minicluster-data/master-0/wal/instance:
uuid: "1524c320cdc84d5a886b598280a7a7ab"
format_stamp: "Formatted at 2026-09-29 19:04:35 on dist-test-slave-lqkv"
server_key: "1ae97fa41683907917110d13ad171669"
server_key_iv: "b3e6a62e33fb0c8ac27bd29db63477b1"
server_key_version: "encryptionkey@0"
I20260929 19:04:35.326431 18981 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 19:04:35.327085 18995 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:04:35.327265 18981 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 19:04:35.327363 18981 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708447806542-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708447806542-392-0/minicluster-data/master-0/wal
uuid: "1524c320cdc84d5a886b598280a7a7ab"
format_stamp: "Formatted at 2026-09-29 19:04:35 on dist-test-slave-lqkv"
server_key: "1ae97fa41683907917110d13ad171669"
server_key_iv: "b3e6a62e33fb0c8ac27bd29db63477b1"
server_key_version: "encryptionkey@0"
I20260929 19:04:35.327446 18981 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708447806542-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708447806542-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708447806542-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:04:35.345985 18981 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:04:35.346232 18981 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:04:35.346397 18981 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:04:35.350198 18981 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37853
I20260929 19:04:35.350229 19047 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37853 every 8 connection(s)
I20260929 19:04:35.350533 18981 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708447806542-392-0/minicluster-data/master-0/data/info.pb
I20260929 19:04:35.351353 19048 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:04:35.353287 19048 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1524c320cdc84d5a886b598280a7a7ab: Bootstrap starting.
I20260929 19:04:35.353703 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUbnLHE/build/release/bin/kudu as pid 18981
I20260929 19:04:35.353782 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708447806542-392-0/minicluster-data/master-0/wal/instance
I20260929 19:04:35.353778 19048 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1524c320cdc84d5a886b598280a7a7ab: Neither blocks nor log segments found. Creating new log.
I20260929 19:04:35.353962 392 external_mini_cluster.cc:1428] Setting key 30c3558e3ca9ba533d3b2739873d3c43
I20260929 19:04:35.354079 19048 log.cc:824] T 00000000000000000000000000000000 P 1524c320cdc84d5a886b598280a7a7ab: Log is configured to *not* fsync() on all Append() calls
I20260929 19:04:35.354761 19048 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1524c320cdc84d5a886b598280a7a7ab: No bootstrap required, opened a new log
I20260929 19:04:35.356586 19048 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1524c320cdc84d5a886b598280a7a7ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1524c320cdc84d5a886b598280a7a7ab" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37853 } }
I20260929 19:04:35.356729 19048 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1524c320cdc84d5a886b598280a7a7ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:04:35.356774 19048 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1524c320cdc84d5a886b598280a7a7ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1524c320cdc84d5a886b598280a7a7ab, State: Initialized, Role: FOLLOWER
I20260929 19:04:35.356940 19048 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1524c320cdc84d5a886b598280a7a7ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1524c320cdc84d5a886b598280a7a7ab" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37853 } }
I20260929 19:04:35.357033 19048 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1524c320cdc84d5a886b598280a7a7ab [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 19:04:35.357074 19048 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1524c320cdc84d5a886b598280a7a7ab [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 19:04:35.357120 19048 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1524c320cdc84d5a886b598280a7a7ab [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:04:35.357640 19048 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1524c320cdc84d5a886b598280a7a7ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1524c320cdc84d5a886b598280a7a7ab" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37853 } }
I20260929 19:04:35.357748 19048 leader_election.cc:302] T 00000000000000000000000000000000 P 1524c320cdc84d5a886b598280a7a7ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1524c320cdc84d5a886b598280a7a7ab; no voters:
I20260929 19:04:35.357924 19048 leader_election.cc:288] T 00000000000000000000000000000000 P 1524c320cdc84d5a886b598280a7a7ab [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 19:04:35.357949 19053 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1524c320cdc84d5a886b598280a7a7ab [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:04:35.358119 19053 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1524c320cdc84d5a886b598280a7a7ab [term 1 LEADER]: Becoming Leader. State: Replica: 1524c320cdc84d5a886b598280a7a7ab, State: Running, Role: LEADER
I20260929 19:04:35.358167 19048 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1524c320cdc84d5a886b598280a7a7ab [sys.catalog]: configured and running, proceeding with master startup.
I20260929 19:04:35.358286 19053 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1524c320cdc84d5a886b598280a7a7ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1524c320cdc84d5a886b598280a7a7ab" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37853 } }
I20260929 19:04:35.358745 19055 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1524c320cdc84d5a886b598280a7a7ab [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1524c320cdc84d5a886b598280a7a7ab. Latest consensus state: current_term: 1 leader_uuid: "1524c320cdc84d5a886b598280a7a7ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1524c320cdc84d5a886b598280a7a7ab" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37853 } } }
I20260929 19:04:35.358837 19055 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1524c320cdc84d5a886b598280a7a7ab [sys.catalog]: This master's current role is: LEADER
I20260929 19:04:35.358959 19054 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1524c320cdc84d5a886b598280a7a7ab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1524c320cdc84d5a886b598280a7a7ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1524c320cdc84d5a886b598280a7a7ab" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37853 } } }
I20260929 19:04:35.359118 19054 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1524c320cdc84d5a886b598280a7a7ab [sys.catalog]: This master's current role is: LEADER
I20260929 19:04:35.359187 19062 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 19:04:35.359633 19062 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 19:04:35.359704 19052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:35.354450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33914 (local address 127.0.98.62:37853)
0929 19:04:35.354629 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0929 19:04:35.354637 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 19:04:35.354662 (+ 25us) server_negotiation.cc:407] Connection header received
0929 19:04:35.355073 (+ 411us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:04:35.355079 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:04:35.355130 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:04:35.355281 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:04:35.355529 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:35.356377 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:35.357433 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:35.357979 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:35.358566 (+ 587us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:04:35.358874 (+ 308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:04:35.358877 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:04:35.358878 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 19:04:35.359078 (+ 200us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:04:35.359114 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 19:04:35.359126 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:04:35.359406 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0929 19:04:35.359434 (+ 28us) server_negotiation.cc:299] Negotiation successful
0929 19:04:35.359508 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":54,"threads_started":1}
I20260929 19:04:35.361111 19062 catalog_manager.cc:1347] Generated new cluster ID: f9c80de7182e47a0b04c09ce4c8af020
I20260929 19:04:35.361162 19062 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 19:04:35.370244 19062 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 19:04:35.370716 19062 catalog_manager.cc:1504] Loading token signing keys...
I20260929 19:04:35.382874 19062 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1524c320cdc84d5a886b598280a7a7ab: Generated new TSK 0
I20260929 19:04:35.383028 19062 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 19:04:35.387326 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUbnLHE/build/release/bin/kudu
/tmp/dist-test-taskUbnLHE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708447806542-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708447806542-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708447806542-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708447806542-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37853
--builtin_ntp_servers=127.0.98.20:38625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 19:04:35.461966 19072 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:04:35.462148 19072 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:04:35.462167 19072 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:04:35.463212 19072 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:04:35.463248 19072 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:04:35.463270 19072 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 19:04:35.464457 19072 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708447806542-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708447806542-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708447806542-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37853
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19072
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708447806542-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 18:09:13 UTC on ccycloud.quasar-hpkalf.root.comops.site
build id 4680913
I20260929 19:04:35.464694 19072 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:04:35.464999 19072 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:04:35.469527 19077 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:04:35.469547 19080 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:04:35.469623 19072 server_base.cc:1034] running on GCE node
W20260929 19:04:35.469554 19078 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:04:35.469935 19072 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:04:35.470201 19072 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:04:35.471345 19072 hybrid_clock.cc:648] HybridClock initialized: now 1790708675471322 us; error 34 us; skew 500 ppm
I20260929 19:04:35.472571 19072 webserver.cc:460] Webserver started at http://127.0.98.1:37337/ using document root <none> and password file <none>
I20260929 19:04:35.472783 19072 fs_manager.cc:359] Metadata directory not provided
I20260929 19:04:35.472901 19072 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:04:35.473044 19072 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:04:35.473923 19072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708447806542-392-0/minicluster-data/ts-0/data/instance:
uuid: "b64d01da0ac04a68ab0b5aa15b66d4d4"
format_stamp: "Formatted at 2026-09-29 19:04:35 on dist-test-slave-lqkv"
server_key: "92e0349dffbff5cc820ad68695cfb284"
server_key_iv: "613a280e04cfbee22c67473732cb3b40"
server_key_version: "encryptionkey@0"
I20260929 19:04:35.474233 19072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708447806542-392-0/minicluster-data/ts-0/wal/instance:
uuid: "b64d01da0ac04a68ab0b5aa15b66d4d4"
format_stamp: "Formatted at 2026-09-29 19:04:35 on dist-test-slave-lqkv"
server_key: "92e0349dffbff5cc820ad68695cfb284"
server_key_iv: "613a280e04cfbee22c67473732cb3b40"
server_key_version: "encryptionkey@0"
I20260929 19:04:35.475385 19072 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 19:04:35.476080 19086 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:04:35.476251 19072 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 19:04:35.476377 19072 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708447806542-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708447806542-392-0/minicluster-data/ts-0/wal
uuid: "b64d01da0ac04a68ab0b5aa15b66d4d4"
format_stamp: "Formatted at 2026-09-29 19:04:35 on dist-test-slave-lqkv"
server_key: "92e0349dffbff5cc820ad68695cfb284"
server_key_iv: "613a280e04cfbee22c67473732cb3b40"
server_key_version: "encryptionkey@0"
I20260929 19:04:35.476459 19072 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708447806542-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708447806542-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708447806542-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:04:35.502058 19072 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:04:35.502328 19072 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:04:35.502471 19072 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:04:35.502669 19072 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:04:35.502938 19072 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:04:35.502987 19072 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:04:35.503024 19072 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:04:35.503047 19072 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:04:35.508292 19072 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42471
I20260929 19:04:35.508329 19199 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42471 every 8 connection(s)
I20260929 19:04:35.508647 19072 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708447806542-392-0/minicluster-data/ts-0/data/info.pb
I20260929 19:04:35.511899 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUbnLHE/build/release/bin/kudu as pid 19072
I20260929 19:04:35.512008 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708447806542-392-0/minicluster-data/ts-0/wal/instance
I20260929 19:04:35.512178 392 external_mini_cluster.cc:1428] Setting key b8ca1eb7d595dfe6a820fcacbfe598ae
I20260929 19:04:35.513096 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUbnLHE/build/release/bin/kudu
/tmp/dist-test-taskUbnLHE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708447806542-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708447806542-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708447806542-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708447806542-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37853
--builtin_ntp_servers=127.0.98.20:38625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 19:04:35.513514 19052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:35.509239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53077 (local address 127.0.98.62:37853)
0929 19:04:35.509275 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0929 19:04:35.509277 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:04:35.509465 (+ 188us) server_negotiation.cc:407] Connection header received
0929 19:04:35.509901 (+ 436us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:04:35.509902 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:04:35.509914 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:04:35.509982 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:04:35.510584 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:35.511032 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:35.511953 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:35.512185 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:35.512254 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:04:35.513172 (+ 918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:04:35.513173 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:04:35.513173 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 19:04:35.513179 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:04:35.513194 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 19:04:35.513199 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:04:35.513256 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0929 19:04:35.513375 (+ 119us) server_negotiation.cc:299] Negotiation successful
0929 19:04:35.513398 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260929 19:04:35.513609 19202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:35.509228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37853 (local address 127.0.98.1:53077)
0929 19:04:35.509366 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0929 19:04:35.509393 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0929 19:04:35.509803 (+ 410us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:04:35.510032 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:04:35.510036 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:04:35.510044 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:04:35.510485 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:04:35.510489 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:04:35.511118 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:04:35.511119 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:04:35.511868 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:04:35.511871 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:04:35.512270 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:04:35.512271 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:04:35.512433 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:04:35.513037 (+ 604us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 19:04:35.513038 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:04:35.513064 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 19:04:35.513084 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:04:35.513282 (+ 198us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:04:35.513288 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:04:35.513292 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 19:04:35.513361 (+ 69us) client_negotiation.cc:240] Negotiation successful
0929 19:04:35.513445 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":84,"thread_start_us":61,"threads_started":1}
I20260929 19:04:35.514051 19200 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37853
I20260929 19:04:35.514158 19200 heartbeater.cc:464] Registering TS with master...
I20260929 19:04:35.514304 19200 heartbeater.cc:511] Master 127.0.98.62:37853 requested a full tablet report, sending...
I20260929 19:04:35.514775 19012 ts_manager.cc:194] Registered new tserver with Master: b64d01da0ac04a68ab0b5aa15b66d4d4 (127.0.98.1:42471)
I20260929 19:04:35.515540 19012 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:53077
W20260929 19:04:35.590081 19203 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:04:35.590252 19203 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:04:35.590274 19203 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:04:35.591288 19203 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:04:35.591328 19203 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:04:35.591351 19203 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 19:04:35.592464 19203 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708447806542-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708447806542-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708447806542-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37853
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19203
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708447806542-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 18:09:13 UTC on ccycloud.quasar-hpkalf.root.comops.site
build id 4680913
I20260929 19:04:35.592617 19203 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:04:35.592869 19203 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:04:35.597532 19208 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:04:35.597568 19203 server_base.cc:1034] running on GCE node
W20260929 19:04:35.597590 19209 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:04:35.597541 19211 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:04:35.597927 19203 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:04:35.598127 19203 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:04:35.599264 19203 hybrid_clock.cc:648] HybridClock initialized: now 1790708675599247 us; error 32 us; skew 500 ppm
I20260929 19:04:35.604560 19203 webserver.cc:460] Webserver started at http://127.0.98.2:34081/ using document root <none> and password file <none>
I20260929 19:04:35.604740 19203 fs_manager.cc:359] Metadata directory not provided
I20260929 19:04:35.604782 19203 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:04:35.604975 19203 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:04:35.605861 19203 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708447806542-392-0/minicluster-data/ts-1/data/instance:
uuid: "8a3d8b9ae51743a1ad0c7673cac70f9b"
format_stamp: "Formatted at 2026-09-29 19:04:35 on dist-test-slave-lqkv"
server_key: "6314aa09a76287b3696716235ba613c4"
server_key_iv: "2810096d8b0c495a6c3e502539f74336"
server_key_version: "encryptionkey@0"
I20260929 19:04:35.606124 19203 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708447806542-392-0/minicluster-data/ts-1/wal/instance:
uuid: "8a3d8b9ae51743a1ad0c7673cac70f9b"
format_stamp: "Formatted at 2026-09-29 19:04:35 on dist-test-slave-lqkv"
server_key: "6314aa09a76287b3696716235ba613c4"
server_key_iv: "2810096d8b0c495a6c3e502539f74336"
server_key_version: "encryptionkey@0"
I20260929 19:04:35.607277 19203 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260929 19:04:35.607941 19217 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:04:35.608126 19203 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 19:04:35.608193 19203 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708447806542-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708447806542-392-0/minicluster-data/ts-1/wal
uuid: "8a3d8b9ae51743a1ad0c7673cac70f9b"
format_stamp: "Formatted at 2026-09-29 19:04:35 on dist-test-slave-lqkv"
server_key: "6314aa09a76287b3696716235ba613c4"
server_key_iv: "2810096d8b0c495a6c3e502539f74336"
server_key_version: "encryptionkey@0"
I20260929 19:04:35.608276 19203 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708447806542-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708447806542-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708447806542-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:04:35.619851 19203 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:04:35.620090 19203 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:04:35.620225 19203 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:04:35.620437 19203 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:04:35.620719 19203 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:04:35.620770 19203 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:04:35.620801 19203 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:04:35.620826 19203 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:04:35.626201 19203 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43009
I20260929 19:04:35.626253 19330 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43009 every 8 connection(s)
I20260929 19:04:35.626537 19203 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708447806542-392-0/minicluster-data/ts-1/data/info.pb
I20260929 19:04:35.628089 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUbnLHE/build/release/bin/kudu as pid 19203
I20260929 19:04:35.628206 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708447806542-392-0/minicluster-data/ts-1/wal/instance
I20260929 19:04:35.628389 392 external_mini_cluster.cc:1428] Setting key 493e80238d48ad99434d3c09718c39ee
I20260929 19:04:35.629457 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUbnLHE/build/release/bin/kudu
/tmp/dist-test-taskUbnLHE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708447806542-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708447806542-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708447806542-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708447806542-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37853
--builtin_ntp_servers=127.0.98.20:38625
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 19:04:35.631968 19052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:35.627151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39673 (local address 127.0.98.62:37853)
0929 19:04:35.627230 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0929 19:04:35.627232 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:04:35.627497 (+ 265us) server_negotiation.cc:407] Connection header received
0929 19:04:35.627934 (+ 437us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:04:35.627935 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:04:35.627949 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:04:35.628022 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:04:35.628735 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:35.629500 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:35.630629 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:35.630828 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:35.630897 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:04:35.631647 (+ 750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:04:35.631649 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:04:35.631649 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 19:04:35.631653 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:04:35.631669 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 19:04:35.631673 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:04:35.631720 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0929 19:04:35.631829 (+ 109us) server_negotiation.cc:299] Negotiation successful
0929 19:04:35.631853 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260929 19:04:35.632033 19333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:35.627137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37853 (local address 127.0.98.2:39673)
0929 19:04:35.627344 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0929 19:04:35.627398 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0929 19:04:35.627832 (+ 434us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:04:35.628071 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:04:35.628074 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:04:35.628081 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:04:35.628660 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:04:35.628665 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:04:35.629598 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:04:35.629600 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:04:35.630526 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:04:35.630528 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:04:35.630924 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:04:35.630925 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:04:35.631041 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:04:35.631511 (+ 470us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 19:04:35.631511 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:04:35.631535 (+ 24us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 19:04:35.631554 (+ 19us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:04:35.631739 (+ 185us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:04:35.631745 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:04:35.631747 (+ 2us) client_negotiation.cc:769] Sending connection context
0929 19:04:35.631804 (+ 57us) client_negotiation.cc:240] Negotiation successful
0929 19:04:35.631876 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":117,"threads_started":1}
I20260929 19:04:35.632512 19331 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37853
I20260929 19:04:35.632606 19331 heartbeater.cc:464] Registering TS with master...
I20260929 19:04:35.632774 19331 heartbeater.cc:511] Master 127.0.98.62:37853 requested a full tablet report, sending...
I20260929 19:04:35.633155 19012 ts_manager.cc:194] Registered new tserver with Master: 8a3d8b9ae51743a1ad0c7673cac70f9b (127.0.98.2:43009)
I20260929 19:04:35.633685 19012 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:39673
W20260929 19:04:35.708338 19334 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:04:35.708520 19334 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:04:35.708540 19334 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:04:35.709581 19334 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:04:35.709614 19334 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:04:35.709637 19334 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 19:04:35.710933 19334 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38625
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708447806542-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708447806542-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708447806542-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37853
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19334
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708447806542-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 18:09:13 UTC on ccycloud.quasar-hpkalf.root.comops.site
build id 4680913
I20260929 19:04:35.711093 19334 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:04:35.711364 19334 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:04:35.716264 19340 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:04:35.716280 19342 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:04:35.716282 19334 server_base.cc:1034] running on GCE node
W20260929 19:04:35.716230 19339 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:04:35.716653 19334 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:04:35.716902 19334 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:04:35.718047 19334 hybrid_clock.cc:648] HybridClock initialized: now 1790708675718022 us; error 41 us; skew 500 ppm
I20260929 19:04:35.719293 19334 webserver.cc:460] Webserver started at http://127.0.98.3:35777/ using document root <none> and password file <none>
I20260929 19:04:35.719534 19334 fs_manager.cc:359] Metadata directory not provided
I20260929 19:04:35.719604 19334 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:04:35.719745 19334 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:04:35.720692 19334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708447806542-392-0/minicluster-data/ts-2/data/instance:
uuid: "68cc675047c0462faaa6d5138ce40d34"
format_stamp: "Formatted at 2026-09-29 19:04:35 on dist-test-slave-lqkv"
server_key: "3da0be6aca17578df3d18e851fd79cc2"
server_key_iv: "525f8f8fe04dd103196b44861a12b328"
server_key_version: "encryptionkey@0"
I20260929 19:04:35.721065 19334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708447806542-392-0/minicluster-data/ts-2/wal/instance:
uuid: "68cc675047c0462faaa6d5138ce40d34"
format_stamp: "Formatted at 2026-09-29 19:04:35 on dist-test-slave-lqkv"
server_key: "3da0be6aca17578df3d18e851fd79cc2"
server_key_iv: "525f8f8fe04dd103196b44861a12b328"
server_key_version: "encryptionkey@0"
I20260929 19:04:35.722268 19334 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 19:04:35.722857 19348 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:04:35.723006 19334 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260929 19:04:35.723088 19334 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708447806542-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708447806542-392-0/minicluster-data/ts-2/wal
uuid: "68cc675047c0462faaa6d5138ce40d34"
format_stamp: "Formatted at 2026-09-29 19:04:35 on dist-test-slave-lqkv"
server_key: "3da0be6aca17578df3d18e851fd79cc2"
server_key_iv: "525f8f8fe04dd103196b44861a12b328"
server_key_version: "encryptionkey@0"
I20260929 19:04:35.723160 19334 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708447806542-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708447806542-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708447806542-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:04:35.729935 19334 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:04:35.730167 19334 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:04:35.730295 19334 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:04:35.730489 19334 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:04:35.730792 19334 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:04:35.730858 19334 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:04:35.730897 19334 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:04:35.730926 19334 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:04:35.736342 19334 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35663
I20260929 19:04:35.736378 19461 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35663 every 8 connection(s)
I20260929 19:04:35.736670 19334 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708447806542-392-0/minicluster-data/ts-2/data/info.pb
I20260929 19:04:35.741647 19052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:35.737353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37811 (local address 127.0.98.62:37853)
0929 19:04:35.737422 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0929 19:04:35.737424 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:04:35.737658 (+ 234us) server_negotiation.cc:407] Connection header received
0929 19:04:35.738118 (+ 460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:04:35.738119 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:04:35.738130 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:04:35.738179 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:04:35.738775 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:35.739255 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:35.740158 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:35.740369 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:35.740451 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:04:35.741296 (+ 845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:04:35.741297 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:04:35.741298 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 19:04:35.741303 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:04:35.741318 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 19:04:35.741322 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:04:35.741410 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0929 19:04:35.741521 (+ 111us) server_negotiation.cc:299] Negotiation successful
0929 19:04:35.741548 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260929 19:04:35.741863 19464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:35.737349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37853 (local address 127.0.98.3:37811)
0929 19:04:35.737529 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0929 19:04:35.737569 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 19:04:35.738016 (+ 447us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:04:35.738251 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:04:35.738255 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:04:35.738262 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:04:35.738687 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:04:35.738691 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:04:35.739345 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:04:35.739346 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:04:35.740069 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:04:35.740071 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:04:35.740461 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:04:35.740462 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:04:35.740580 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:04:35.741086 (+ 506us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 19:04:35.741087 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:04:35.741111 (+ 24us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 19:04:35.741129 (+ 18us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:04:35.741418 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:04:35.741425 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:04:35.741428 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 19:04:35.741517 (+ 89us) client_negotiation.cc:240] Negotiation successful
0929 19:04:35.741637 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":87,"threads_started":1}
I20260929 19:04:35.742383 19462 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37853
I20260929 19:04:35.742451 19462 heartbeater.cc:464] Registering TS with master...
I20260929 19:04:35.742580 19462 heartbeater.cc:511] Master 127.0.98.62:37853 requested a full tablet report, sending...
I20260929 19:04:35.742858 19012 ts_manager.cc:194] Registered new tserver with Master: 68cc675047c0462faaa6d5138ce40d34 (127.0.98.3:35663)
I20260929 19:04:35.743186 19012 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:37811
I20260929 19:04:35.744516 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUbnLHE/build/release/bin/kudu as pid 19334
I20260929 19:04:35.744603 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708447806542-392-0/minicluster-data/ts-2/wal/instance
I20260929 19:04:35.744740 392 external_mini_cluster.cc:1428] Setting key 178a9440e03d7da7d9fba4af35fdb6e8
I20260929 19:04:35.745631 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 19:04:35.749701 19052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:35.746884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33924 (local address 127.0.98.62:37853)
0929 19:04:35.746960 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0929 19:04:35.746961 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 19:04:35.747062 (+ 101us) server_negotiation.cc:407] Connection header received
0929 19:04:35.747135 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:04:35.747136 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:04:35.747146 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:04:35.747196 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:04:35.747385 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:35.747827 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:35.748573 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:35.748773 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:35.748893 (+ 120us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:04:35.749437 (+ 544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:04:35.749438 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:04:35.749439 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 19:04:35.749443 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:04:35.749466 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 19:04:35.749469 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:04:35.749565 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0929 19:04:35.749623 (+ 58us) server_negotiation.cc:299] Negotiation successful
0929 19:04:35.749632 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 19:04:35.751120 19012 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33924:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 19:04:35.752632 19012 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 19:04:35.758584 19475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:35.754701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38414 (local address 127.0.98.1:42471)
0929 19:04:35.754869 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0929 19:04:35.754873 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 19:04:35.754923 (+ 50us) server_negotiation.cc:407] Connection header received
0929 19:04:35.755014 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:04:35.755016 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:04:35.755031 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:04:35.755099 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:04:35.755267 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:35.756310 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:35.757549 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:35.758237 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:35.758289 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:04:35.758356 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0929 19:04:35.758477 (+ 121us) server_negotiation.cc:299] Negotiation successful
0929 19:04:35.758509 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":78,"threads_started":1}
I20260929 19:04:35.758590 19474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:35.754605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42471 (local address 127.0.0.1:38414)
0929 19:04:35.754865 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0929 19:04:35.754883 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 19:04:35.754932 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:04:35.755159 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:04:35.755160 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:04:35.755161 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:04:35.755198 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:04:35.755199 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:04:35.756385 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:04:35.756386 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:04:35.757458 (+ 1072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:04:35.757460 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:04:35.758312 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:04:35.758312 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:04:35.758368 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:04:35.758394 (+ 26us) client_negotiation.cc:769] Sending connection context
0929 19:04:35.758451 (+ 57us) client_negotiation.cc:240] Negotiation successful
0929 19:04:35.758494 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"mutex_wait_us":3,"thread_start_us":60,"threads_started":1}
I20260929 19:04:35.758817 19473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:35.754585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54138 (local address 127.0.98.3:35663)
0929 19:04:35.755027 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0929 19:04:35.755031 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 19:04:35.755038 (+ 7us) server_negotiation.cc:407] Connection header received
0929 19:04:35.755065 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:04:35.755066 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:04:35.755078 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:04:35.755139 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:04:35.755234 (+ 95us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:35.756038 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:35.757462 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:35.758131 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:35.758192 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:04:35.758265 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 19:04:35.758727 (+ 462us) server_negotiation.cc:299] Negotiation successful
0929 19:04:35.758762 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":415,"thread_start_us":69,"threads_started":1}
I20260929 19:04:35.758589 19471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:35.754439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35663 (local address 127.0.0.1:54138)
0929 19:04:35.754756 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0929 19:04:35.754764 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0929 19:04:35.754830 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:04:35.755139 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:04:35.755139 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:04:35.755141 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:04:35.755187 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:04:35.755188 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:04:35.756113 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:04:35.756115 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:04:35.757341 (+ 1226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:04:35.757343 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:04:35.758297 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:04:35.758298 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:04:35.758381 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:04:35.758393 (+ 12us) client_negotiation.cc:769] Sending connection context
0929 19:04:35.758471 (+ 78us) client_negotiation.cc:240] Negotiation successful
0929 19:04:35.758511 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"mutex_wait_us":4,"thread_start_us":51,"threads_started":1}
I20260929 19:04:35.759373 19470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:35.754439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43009 (local address 127.0.0.1:39556)
0929 19:04:35.754715 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0929 19:04:35.754749 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0929 19:04:35.754810 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:04:35.755042 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:04:35.755045 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:04:35.755047 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:04:35.755104 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:04:35.755106 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:04:35.757541 (+ 2435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:04:35.757541 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:04:35.758299 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:04:35.758302 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:04:35.759182 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:04:35.759183 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:04:35.759238 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:04:35.759241 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 19:04:35.759287 (+ 46us) client_negotiation.cc:240] Negotiation successful
0929 19:04:35.759320 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":66,"threads_started":1}
I20260929 19:04:35.759264 19134 tablet_service.cc:1467] Processing CreateTablet for tablet c87a1087abd74204bb968f7e12e74640 (DEFAULT_TABLE table=test-table [id=b19afbd3d4414795b6bc6bfb6a730a21]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:04:35.759595 19134 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c87a1087abd74204bb968f7e12e74640. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:04:35.759629 19472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:35.754533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39556 (local address 127.0.98.2:43009)
0929 19:04:35.754700 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0929 19:04:35.754704 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 19:04:35.754886 (+ 182us) server_negotiation.cc:407] Connection header received
0929 19:04:35.754932 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:04:35.754935 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:04:35.754949 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:04:35.755024 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:04:35.755186 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:35.756297 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:35.758377 (+ 2080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:35.759008 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:35.759064 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:04:35.759151 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0929 19:04:35.759507 (+ 356us) server_negotiation.cc:299] Negotiation successful
0929 19:04:35.759547 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":71,"threads_started":1}
I20260929 19:04:35.759704 19396 tablet_service.cc:1467] Processing CreateTablet for tablet c87a1087abd74204bb968f7e12e74640 (DEFAULT_TABLE table=test-table [id=b19afbd3d4414795b6bc6bfb6a730a21]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:04:35.759990 19396 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c87a1087abd74204bb968f7e12e74640. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:04:35.760143 19265 tablet_service.cc:1467] Processing CreateTablet for tablet c87a1087abd74204bb968f7e12e74640 (DEFAULT_TABLE table=test-table [id=b19afbd3d4414795b6bc6bfb6a730a21]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:04:35.760421 19265 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c87a1087abd74204bb968f7e12e74640. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:04:35.761986 19476 tablet_bootstrap.cc:492] T c87a1087abd74204bb968f7e12e74640 P b64d01da0ac04a68ab0b5aa15b66d4d4: Bootstrap starting.
I20260929 19:04:35.762634 19476 tablet_bootstrap.cc:654] T c87a1087abd74204bb968f7e12e74640 P b64d01da0ac04a68ab0b5aa15b66d4d4: Neither blocks nor log segments found. Creating new log.
I20260929 19:04:35.762894 19477 tablet_bootstrap.cc:492] T c87a1087abd74204bb968f7e12e74640 P 68cc675047c0462faaa6d5138ce40d34: Bootstrap starting.
I20260929 19:04:35.762954 19476 log.cc:824] T c87a1087abd74204bb968f7e12e74640 P b64d01da0ac04a68ab0b5aa15b66d4d4: Log is configured to *not* fsync() on all Append() calls
I20260929 19:04:35.763530 19477 tablet_bootstrap.cc:654] T c87a1087abd74204bb968f7e12e74640 P 68cc675047c0462faaa6d5138ce40d34: Neither blocks nor log segments found. Creating new log.
I20260929 19:04:35.763638 19476 tablet_bootstrap.cc:492] T c87a1087abd74204bb968f7e12e74640 P b64d01da0ac04a68ab0b5aa15b66d4d4: No bootstrap required, opened a new log
I20260929 19:04:35.763705 19476 ts_tablet_manager.cc:1397] T c87a1087abd74204bb968f7e12e74640 P b64d01da0ac04a68ab0b5aa15b66d4d4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 19:04:35.763819 19477 log.cc:824] T c87a1087abd74204bb968f7e12e74640 P 68cc675047c0462faaa6d5138ce40d34: Log is configured to *not* fsync() on all Append() calls
I20260929 19:04:35.764461 19477 tablet_bootstrap.cc:492] T c87a1087abd74204bb968f7e12e74640 P 68cc675047c0462faaa6d5138ce40d34: No bootstrap required, opened a new log
I20260929 19:04:35.764524 19477 ts_tablet_manager.cc:1397] T c87a1087abd74204bb968f7e12e74640 P 68cc675047c0462faaa6d5138ce40d34: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 19:04:35.764912 19479 tablet_bootstrap.cc:492] T c87a1087abd74204bb968f7e12e74640 P 8a3d8b9ae51743a1ad0c7673cac70f9b: Bootstrap starting.
I20260929 19:04:35.765476 19479 tablet_bootstrap.cc:654] T c87a1087abd74204bb968f7e12e74640 P 8a3d8b9ae51743a1ad0c7673cac70f9b: Neither blocks nor log segments found. Creating new log.
I20260929 19:04:35.765759 19479 log.cc:824] T c87a1087abd74204bb968f7e12e74640 P 8a3d8b9ae51743a1ad0c7673cac70f9b: Log is configured to *not* fsync() on all Append() calls
I20260929 19:04:35.766144 19476 raft_consensus.cc:348] T c87a1087abd74204bb968f7e12e74640 P b64d01da0ac04a68ab0b5aa15b66d4d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a3d8b9ae51743a1ad0c7673cac70f9b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43009 } } peers { permanent_uuid: "68cc675047c0462faaa6d5138ce40d34" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35663 } } peers { permanent_uuid: "b64d01da0ac04a68ab0b5aa15b66d4d4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42471 } }
I20260929 19:04:35.766292 19476 raft_consensus.cc:374] T c87a1087abd74204bb968f7e12e74640 P b64d01da0ac04a68ab0b5aa15b66d4d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:04:35.766345 19476 raft_consensus.cc:729] T c87a1087abd74204bb968f7e12e74640 P b64d01da0ac04a68ab0b5aa15b66d4d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b64d01da0ac04a68ab0b5aa15b66d4d4, State: Initialized, Role: FOLLOWER
I20260929 19:04:35.766428 19479 tablet_bootstrap.cc:492] T c87a1087abd74204bb968f7e12e74640 P 8a3d8b9ae51743a1ad0c7673cac70f9b: No bootstrap required, opened a new log
I20260929 19:04:35.766491 19479 ts_tablet_manager.cc:1397] T c87a1087abd74204bb968f7e12e74640 P 8a3d8b9ae51743a1ad0c7673cac70f9b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 19:04:35.766465 19476 consensus_queue.cc:260] T c87a1087abd74204bb968f7e12e74640 P b64d01da0ac04a68ab0b5aa15b66d4d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a3d8b9ae51743a1ad0c7673cac70f9b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43009 } } peers { permanent_uuid: "68cc675047c0462faaa6d5138ce40d34" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35663 } } peers { permanent_uuid: "b64d01da0ac04a68ab0b5aa15b66d4d4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42471 } }
I20260929 19:04:35.766731 19200 heartbeater.cc:503] Master 127.0.98.62:37853 was elected leader, sending a full tablet report...
I20260929 19:04:35.766944 19476 ts_tablet_manager.cc:1428] T c87a1087abd74204bb968f7e12e74640 P b64d01da0ac04a68ab0b5aa15b66d4d4: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 19:04:35.767174 19477 raft_consensus.cc:348] T c87a1087abd74204bb968f7e12e74640 P 68cc675047c0462faaa6d5138ce40d34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a3d8b9ae51743a1ad0c7673cac70f9b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43009 } } peers { permanent_uuid: "68cc675047c0462faaa6d5138ce40d34" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35663 } } peers { permanent_uuid: "b64d01da0ac04a68ab0b5aa15b66d4d4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42471 } }
I20260929 19:04:35.767330 19477 raft_consensus.cc:374] T c87a1087abd74204bb968f7e12e74640 P 68cc675047c0462faaa6d5138ce40d34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:04:35.767386 19477 raft_consensus.cc:729] T c87a1087abd74204bb968f7e12e74640 P 68cc675047c0462faaa6d5138ce40d34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68cc675047c0462faaa6d5138ce40d34, State: Initialized, Role: FOLLOWER
I20260929 19:04:35.767503 19477 consensus_queue.cc:260] T c87a1087abd74204bb968f7e12e74640 P 68cc675047c0462faaa6d5138ce40d34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a3d8b9ae51743a1ad0c7673cac70f9b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43009 } } peers { permanent_uuid: "68cc675047c0462faaa6d5138ce40d34" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35663 } } peers { permanent_uuid: "b64d01da0ac04a68ab0b5aa15b66d4d4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42471 } }
I20260929 19:04:35.767752 19477 ts_tablet_manager.cc:1428] T c87a1087abd74204bb968f7e12e74640 P 68cc675047c0462faaa6d5138ce40d34: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260929 19:04:35.767784 19462 heartbeater.cc:503] Master 127.0.98.62:37853 was elected leader, sending a full tablet report...
I20260929 19:04:35.768998 19479 raft_consensus.cc:348] T c87a1087abd74204bb968f7e12e74640 P 8a3d8b9ae51743a1ad0c7673cac70f9b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a3d8b9ae51743a1ad0c7673cac70f9b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43009 } } peers { permanent_uuid: "68cc675047c0462faaa6d5138ce40d34" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35663 } } peers { permanent_uuid: "b64d01da0ac04a68ab0b5aa15b66d4d4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42471 } }
I20260929 19:04:35.769127 19479 raft_consensus.cc:374] T c87a1087abd74204bb968f7e12e74640 P 8a3d8b9ae51743a1ad0c7673cac70f9b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:04:35.769179 19479 raft_consensus.cc:729] T c87a1087abd74204bb968f7e12e74640 P 8a3d8b9ae51743a1ad0c7673cac70f9b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a3d8b9ae51743a1ad0c7673cac70f9b, State: Initialized, Role: FOLLOWER
I20260929 19:04:35.769272 19479 consensus_queue.cc:260] T c87a1087abd74204bb968f7e12e74640 P 8a3d8b9ae51743a1ad0c7673cac70f9b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a3d8b9ae51743a1ad0c7673cac70f9b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43009 } } peers { permanent_uuid: "68cc675047c0462faaa6d5138ce40d34" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35663 } } peers { permanent_uuid: "b64d01da0ac04a68ab0b5aa15b66d4d4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42471 } }
I20260929 19:04:35.769472 19479 ts_tablet_manager.cc:1428] T c87a1087abd74204bb968f7e12e74640 P 8a3d8b9ae51743a1ad0c7673cac70f9b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 19:04:35.769546 19331 heartbeater.cc:503] Master 127.0.98.62:37853 was elected leader, sending a full tablet report...
W20260929 19:04:35.876992 19332 tablet.cc:2369] T c87a1087abd74204bb968f7e12e74640 P 8a3d8b9ae51743a1ad0c7673cac70f9b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 19:04:35.987205 19463 tablet.cc:2369] T c87a1087abd74204bb968f7e12e74640 P 68cc675047c0462faaa6d5138ce40d34: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 19:04:35.999418 19482 raft_consensus.cc:482] T c87a1087abd74204bb968f7e12e74640 P b64d01da0ac04a68ab0b5aa15b66d4d4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 19:04:35.999532 19482 raft_consensus.cc:504] T c87a1087abd74204bb968f7e12e74640 P b64d01da0ac04a68ab0b5aa15b66d4d4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a3d8b9ae51743a1ad0c7673cac70f9b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43009 } } peers { permanent_uuid: "68cc675047c0462faaa6d5138ce40d34" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35663 } } peers { permanent_uuid: "b64d01da0ac04a68ab0b5aa15b66d4d4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42471 } }
I20260929 19:04:35.999823 19482 leader_election.cc:288] T c87a1087abd74204bb968f7e12e74640 P b64d01da0ac04a68ab0b5aa15b66d4d4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a3d8b9ae51743a1ad0c7673cac70f9b (127.0.98.2:43009), 68cc675047c0462faaa6d5138ce40d34 (127.0.98.3:35663)
I20260929 19:04:36.002480 19473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:36.000069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42535 (local address 127.0.98.3:35663)
0929 19:04:36.000127 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0929 19:04:36.000129 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:04:36.000139 (+ 10us) server_negotiation.cc:407] Connection header received
0929 19:04:36.000172 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:04:36.000173 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:04:36.000183 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:04:36.000247 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:04:36.000451 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:36.001011 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:36.001873 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:36.002180 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:36.002231 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:04:36.002238 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0929 19:04:36.002378 (+ 140us) server_negotiation.cc:299] Negotiation successful
0929 19:04:36.002419 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260929 19:04:36.002480 19202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:35.999992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35663 (local address 127.0.98.1:42535)
0929 19:04:36.000048 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0929 19:04:36.000058 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 19:04:36.000115 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:04:36.000278 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:04:36.000279 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:04:36.000281 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:04:36.000359 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:04:36.000362 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:04:36.001078 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:04:36.001079 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:04:36.001783 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:04:36.001786 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:04:36.002252 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:04:36.002253 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:04:36.002315 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:04:36.002318 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 19:04:36.002361 (+ 43us) client_negotiation.cc:240] Negotiation successful
0929 19:04:36.002396 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260929 19:04:36.002576 19472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:36.000148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55287 (local address 127.0.98.2:43009)
0929 19:04:36.000252 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0929 19:04:36.000254 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:04:36.000265 (+ 11us) server_negotiation.cc:407] Connection header received
0929 19:04:36.000340 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:04:36.000341 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:04:36.000351 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:04:36.000423 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:04:36.000583 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:36.001074 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:36.001900 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:36.002202 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:36.002292 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:04:36.002300 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0929 19:04:36.002455 (+ 155us) server_negotiation.cc:299] Negotiation successful
0929 19:04:36.002496 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 19:04:36.002780 19485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:36.000006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43009 (local address 127.0.98.1:55287)
0929 19:04:36.000214 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0929 19:04:36.000224 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 19:04:36.000272 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:04:36.000447 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:04:36.000448 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:04:36.000450 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:04:36.000515 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:04:36.000516 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:04:36.001145 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:04:36.001146 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:04:36.001803 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:04:36.001805 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:04:36.002309 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:04:36.002310 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:04:36.002387 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:04:36.002390 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 19:04:36.002437 (+ 47us) client_negotiation.cc:240] Negotiation successful
0929 19:04:36.002467 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":64,"threads_started":1}
I20260929 19:04:36.002820 19416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c87a1087abd74204bb968f7e12e74640" candidate_uuid: "b64d01da0ac04a68ab0b5aa15b66d4d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68cc675047c0462faaa6d5138ce40d34" is_pre_election: true
I20260929 19:04:36.003013 19416 raft_consensus.cc:2454] T c87a1087abd74204bb968f7e12e74640 P 68cc675047c0462faaa6d5138ce40d34 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b64d01da0ac04a68ab0b5aa15b66d4d4 in term 0.
I20260929 19:04:36.003095 19285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c87a1087abd74204bb968f7e12e74640" candidate_uuid: "b64d01da0ac04a68ab0b5aa15b66d4d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a3d8b9ae51743a1ad0c7673cac70f9b" is_pre_election: true
I20260929 19:04:36.003232 19285 raft_consensus.cc:2454] T c87a1087abd74204bb968f7e12e74640 P 8a3d8b9ae51743a1ad0c7673cac70f9b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b64d01da0ac04a68ab0b5aa15b66d4d4 in term 0.
I20260929 19:04:36.003232 19087 leader_election.cc:302] T c87a1087abd74204bb968f7e12e74640 P b64d01da0ac04a68ab0b5aa15b66d4d4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68cc675047c0462faaa6d5138ce40d34, b64d01da0ac04a68ab0b5aa15b66d4d4; no voters:
I20260929 19:04:36.003372 19482 raft_consensus.cc:2781] T c87a1087abd74204bb968f7e12e74640 P b64d01da0ac04a68ab0b5aa15b66d4d4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 19:04:36.003424 19482 raft_consensus.cc:482] T c87a1087abd74204bb968f7e12e74640 P b64d01da0ac04a68ab0b5aa15b66d4d4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 19:04:36.003448 19482 raft_consensus.cc:3037] T c87a1087abd74204bb968f7e12e74640 P b64d01da0ac04a68ab0b5aa15b66d4d4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:04:36.004086 19482 raft_consensus.cc:504] T c87a1087abd74204bb968f7e12e74640 P b64d01da0ac04a68ab0b5aa15b66d4d4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a3d8b9ae51743a1ad0c7673cac70f9b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43009 } } peers { permanent_uuid: "68cc675047c0462faaa6d5138ce40d34" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35663 } } peers { permanent_uuid: "b64d01da0ac04a68ab0b5aa15b66d4d4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42471 } }
I20260929 19:04:36.004217 19482 leader_election.cc:288] T c87a1087abd74204bb968f7e12e74640 P b64d01da0ac04a68ab0b5aa15b66d4d4 [CANDIDATE]: Term 1 election: Requested vote from peers 8a3d8b9ae51743a1ad0c7673cac70f9b (127.0.98.2:43009), 68cc675047c0462faaa6d5138ce40d34 (127.0.98.3:35663)
I20260929 19:04:36.004398 19416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c87a1087abd74204bb968f7e12e74640" candidate_uuid: "b64d01da0ac04a68ab0b5aa15b66d4d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68cc675047c0462faaa6d5138ce40d34"
I20260929 19:04:36.004392 19285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c87a1087abd74204bb968f7e12e74640" candidate_uuid: "b64d01da0ac04a68ab0b5aa15b66d4d4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a3d8b9ae51743a1ad0c7673cac70f9b"
I20260929 19:04:36.004469 19416 raft_consensus.cc:3037] T c87a1087abd74204bb968f7e12e74640 P 68cc675047c0462faaa6d5138ce40d34 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:04:36.004469 19285 raft_consensus.cc:3037] T c87a1087abd74204bb968f7e12e74640 P 8a3d8b9ae51743a1ad0c7673cac70f9b [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:04:36.005306 19416 raft_consensus.cc:2454] T c87a1087abd74204bb968f7e12e74640 P 68cc675047c0462faaa6d5138ce40d34 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b64d01da0ac04a68ab0b5aa15b66d4d4 in term 1.
I20260929 19:04:36.005306 19285 raft_consensus.cc:2454] T c87a1087abd74204bb968f7e12e74640 P 8a3d8b9ae51743a1ad0c7673cac70f9b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b64d01da0ac04a68ab0b5aa15b66d4d4 in term 1.
I20260929 19:04:36.005465 19089 leader_election.cc:302] T c87a1087abd74204bb968f7e12e74640 P b64d01da0ac04a68ab0b5aa15b66d4d4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a3d8b9ae51743a1ad0c7673cac70f9b, b64d01da0ac04a68ab0b5aa15b66d4d4; no voters:
I20260929 19:04:36.005551 19482 raft_consensus.cc:2781] T c87a1087abd74204bb968f7e12e74640 P b64d01da0ac04a68ab0b5aa15b66d4d4 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:04:36.005679 19482 raft_consensus.cc:686] T c87a1087abd74204bb968f7e12e74640 P b64d01da0ac04a68ab0b5aa15b66d4d4 [term 1 LEADER]: Becoming Leader. State: Replica: b64d01da0ac04a68ab0b5aa15b66d4d4, State: Running, Role: LEADER
I20260929 19:04:36.005764 19482 consensus_queue.cc:237] T c87a1087abd74204bb968f7e12e74640 P b64d01da0ac04a68ab0b5aa15b66d4d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a3d8b9ae51743a1ad0c7673cac70f9b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43009 } } peers { permanent_uuid: "68cc675047c0462faaa6d5138ce40d34" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35663 } } peers { permanent_uuid: "b64d01da0ac04a68ab0b5aa15b66d4d4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42471 } }
I20260929 19:04:36.006484 19011 catalog_manager.cc:5495] T c87a1087abd74204bb968f7e12e74640 P b64d01da0ac04a68ab0b5aa15b66d4d4 reported cstate change: term changed from 0 to 1, leader changed from <none> to b64d01da0ac04a68ab0b5aa15b66d4d4 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "b64d01da0ac04a68ab0b5aa15b66d4d4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a3d8b9ae51743a1ad0c7673cac70f9b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43009 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "68cc675047c0462faaa6d5138ce40d34" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35663 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b64d01da0ac04a68ab0b5aa15b66d4d4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42471 } health_report { overall_health: HEALTHY } } }
W20260929 19:04:36.009302 19201 tablet.cc:2369] T c87a1087abd74204bb968f7e12e74640 P b64d01da0ac04a68ab0b5aa15b66d4d4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 19:04:36.041776 19475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:36.039488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38428 (local address 127.0.98.1:42471)
0929 19:04:36.039551 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0929 19:04:36.039554 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 19:04:36.039563 (+ 9us) server_negotiation.cc:407] Connection header received
0929 19:04:36.039597 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:04:36.039597 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:04:36.039623 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:04:36.039697 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 19:04:36.039843 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:36.040293 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:36.041051 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:36.041239 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:36.041313 (+ 74us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:04:36.041446 (+ 133us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 19:04:36.041531 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 19:04:36.041636 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0929 19:04:36.041667 (+ 31us) server_negotiation.cc:299] Negotiation successful
0929 19:04:36.041677 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260929 19:04:36.043377 19416 raft_consensus.cc:1261] T c87a1087abd74204bb968f7e12e74640 P 68cc675047c0462faaa6d5138ce40d34 [term 1 FOLLOWER]: Refusing update from remote peer b64d01da0ac04a68ab0b5aa15b66d4d4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 19:04:36.043385 19285 raft_consensus.cc:1261] T c87a1087abd74204bb968f7e12e74640 P 8a3d8b9ae51743a1ad0c7673cac70f9b [term 1 FOLLOWER]: Refusing update from remote peer b64d01da0ac04a68ab0b5aa15b66d4d4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 19:04:36.043744 19486 consensus_queue.cc:1035] T c87a1087abd74204bb968f7e12e74640 P b64d01da0ac04a68ab0b5aa15b66d4d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68cc675047c0462faaa6d5138ce40d34" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 19:04:36.043895 19482 consensus_queue.cc:1035] T c87a1087abd74204bb968f7e12e74640 P b64d01da0ac04a68ab0b5aa15b66d4d4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a3d8b9ae51743a1ad0c7673cac70f9b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 19:04:36.045910 19489 mvcc.cc:205] Tried to move back new op lower bound from 7334742737071329280 to 7334742736920150016. Current Snapshot: MvccSnapshot[applied={T|T < 7334742737071329280}]
I20260929 19:04:36.046273 19490 mvcc.cc:205] Tried to move back new op lower bound from 7334742737071329280 to 7334742736920150016. Current Snapshot: MvccSnapshot[applied={T|T < 7334742737071329280}]
I20260929 19:04:36.046818 19491 mvcc.cc:205] Tried to move back new op lower bound from 7334742737071329280 to 7334742736920150016. Current Snapshot: MvccSnapshot[applied={T|T < 7334742737071329280}]
I20260929 19:04:36.048348 19011 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33924:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 19:04:36.048456 19011 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33924:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 19:04:36.049175 19011 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1524c320cdc84d5a886b598280a7a7ab: Sending DeleteTablet for 3 replicas of tablet c87a1087abd74204bb968f7e12e74640
I20260929 19:04:36.049445 19265 tablet_service.cc:1514] Processing DeleteTablet for tablet c87a1087abd74204bb968f7e12e74640 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:04:36 UTC) from {username='slave'} at 127.0.0.1:39556
I20260929 19:04:36.049531 19134 tablet_service.cc:1514] Processing DeleteTablet for tablet c87a1087abd74204bb968f7e12e74640 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:04:36 UTC) from {username='slave'} at 127.0.0.1:38414
I20260929 19:04:36.049728 19500 tablet_replica.cc:330] T c87a1087abd74204bb968f7e12e74640 P b64d01da0ac04a68ab0b5aa15b66d4d4: stopping tablet replica
I20260929 19:04:36.049815 19500 raft_consensus.cc:2229] T c87a1087abd74204bb968f7e12e74640 P b64d01da0ac04a68ab0b5aa15b66d4d4 [term 1 LEADER]: Raft consensus shutting down.
I20260929 19:04:36.049809 19396 tablet_service.cc:1514] Processing DeleteTablet for tablet c87a1087abd74204bb968f7e12e74640 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:04:36 UTC) from {username='slave'} at 127.0.0.1:54138
I20260929 19:04:36.049919 19500 raft_consensus.cc:2258] T c87a1087abd74204bb968f7e12e74640 P b64d01da0ac04a68ab0b5aa15b66d4d4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 19:04:36.050035 19501 tablet_replica.cc:330] T c87a1087abd74204bb968f7e12e74640 P 68cc675047c0462faaa6d5138ce40d34: stopping tablet replica
I20260929 19:04:36.050115 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUbnLHE/build/release/bin/kudu with pid 19072
I20260929 19:04:36.050336 19500 ts_tablet_manager.cc:1905] T c87a1087abd74204bb968f7e12e74640 P b64d01da0ac04a68ab0b5aa15b66d4d4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 19:04:36.050890 19501 raft_consensus.cc:2229] T c87a1087abd74204bb968f7e12e74640 P 68cc675047c0462faaa6d5138ce40d34 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 19:04:36.051049 19501 raft_consensus.cc:2258] T c87a1087abd74204bb968f7e12e74640 P 68cc675047c0462faaa6d5138ce40d34 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 19:04:36.051406 19501 ts_tablet_manager.cc:1905] T c87a1087abd74204bb968f7e12e74640 P 68cc675047c0462faaa6d5138ce40d34: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 19:04:36.052875 19501 ts_tablet_manager.cc:1918] T c87a1087abd74204bb968f7e12e74640 P 68cc675047c0462faaa6d5138ce40d34: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 19:04:36.052963 19501 log.cc:1196] T c87a1087abd74204bb968f7e12e74640 P 68cc675047c0462faaa6d5138ce40d34: Deleting WAL directory at /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790708447806542-392-0/minicluster-data/ts-2/wal/wals/c87a1087abd74204bb968f7e12e74640
I20260929 19:04:36.053191 19501 ts_tablet_manager.cc:1939] T c87a1087abd74204bb968f7e12e74640 P 68cc675047c0462faaa6d5138ce40d34: Deleting consensus metadata
I20260929 19:04:36.053565 18996 catalog_manager.cc:4841] TS 68cc675047c0462faaa6d5138ce40d34 (127.0.98.3:35663): tablet c87a1087abd74204bb968f7e12e74640 (table test-table [id=b19afbd3d4414795b6bc6bfb6a730a21]) successfully deleted
W20260929 19:04:36.056106 18996 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:42471 (error 108)
I20260929 19:04:36.056360 19499 tablet_replica.cc:330] T c87a1087abd74204bb968f7e12e74640 P 8a3d8b9ae51743a1ad0c7673cac70f9b: stopping tablet replica
I20260929 19:04:36.056444 19499 raft_consensus.cc:2229] T c87a1087abd74204bb968f7e12e74640 P 8a3d8b9ae51743a1ad0c7673cac70f9b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 19:04:36.056548 19499 raft_consensus.cc:2258] T c87a1087abd74204bb968f7e12e74640 P 8a3d8b9ae51743a1ad0c7673cac70f9b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 19:04:36.056569 19474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:36.056466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42471 (local address 127.0.0.1:38442)
0929 19:04:36.056504 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0929 19:04:36.056538 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
W20260929 19:04:36.056672 18996 catalog_manager.cc:4584] TS b64d01da0ac04a68ab0b5aa15b66d4d4 (127.0.98.1:42471): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c87a1087abd74204bb968f7e12e74640: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42471: connect: Connection refused (error 111)
I20260929 19:04:36.056749 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUbnLHE/build/release/bin/kudu with pid 19203
I20260929 19:04:36.062275 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUbnLHE/build/release/bin/kudu with pid 19334
I20260929 19:04:36.062322 19474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:36.062232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43009 (local address 127.0.0.1:39560)
0929 19:04:36.062277 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0929 19:04:36.062304 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:43009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260929 19:04:36.066125 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUbnLHE/build/release/bin/kudu with pid 18981
2026-09-29T19:04:36Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (850 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708447806542-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 19:04:36 dist-test-slave-lqkv krb5kdc[19505](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 19:04:36 dist-test-slave-lqkv krb5kdc[19505](info): set up 2 sockets
Sep 29 19:04:36 dist-test-slave-lqkv krb5kdc[19505](info): commencing operation
krb5kdc: starting...
W20260929 19:04:37.036489 392 mini_kdc.cc:121] Time spent starting KDC: real 0.952s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708447806542-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708447806542-392-0/krb5kdc/test-user.keytab.
Sep 29 19:04:37 dist-test-slave-lqkv krb5kdc[19505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708677, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T19:04:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T19:04:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708447806542-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708447806542-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 19:04:37.139616 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUbnLHE/build/release/bin/kudu
/tmp/dist-test-taskUbnLHE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708447806542-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708447806542-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708447806542-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708447806542-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43741
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708447806542-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43741
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708447806542-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708447806542-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708447806542-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:04:37.212993 19518 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:04:37.213192 19518 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:04:37.213225 19518 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:04:37.214305 19518 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 19:04:37.214367 19518 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:04:37.214416 19518 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:04:37.214447 19518 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 19:04:37.214469 19518 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 19:04:37.215817 19518 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708447806542-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708447806542-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43741
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708447806542-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43741
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708447806542-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19518
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708447806542-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 18:09:13 UTC on ccycloud.quasar-hpkalf.root.comops.site
build id 4680913
I20260929 19:04:37.216054 19518 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:04:37.216249 19518 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:04:37.220705 19524 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:04:37.220765 19523 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:04:37.220921 19518 server_base.cc:1034] running on GCE node
W20260929 19:04:37.220992 19526 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:04:37.221207 19518 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:04:37.221481 19518 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:04:37.222615 19518 hybrid_clock.cc:648] HybridClock initialized: now 1790708677222605 us; error 26 us; skew 500 ppm
Sep 29 19:04:37 dist-test-slave-lqkv krb5kdc[19505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708677, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:04:37.224997 19518 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 19:04:37.225411 19518 webserver.cc:460] Webserver started at http://127.0.98.62:44807/ using document root <none> and password file <none>
I20260929 19:04:37.225610 19518 fs_manager.cc:359] Metadata directory not provided
I20260929 19:04:37.225672 19518 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:04:37.225821 19518 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:04:37.226758 19518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708447806542-392-0/minicluster-data/master-0/data/instance:
uuid: "3d12b61530af4068bc771aa9c1ad7316"
format_stamp: "Formatted at 2026-09-29 19:04:37 on dist-test-slave-lqkv"
server_key: "f81da2a074bc060c025e60b487172242"
server_key_iv: "99cb04ba3871a26491727be13795bdcc"
server_key_version: "encryptionkey@0"
I20260929 19:04:37.227105 19518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708447806542-392-0/minicluster-data/master-0/wal/instance:
uuid: "3d12b61530af4068bc771aa9c1ad7316"
format_stamp: "Formatted at 2026-09-29 19:04:37 on dist-test-slave-lqkv"
server_key: "f81da2a074bc060c025e60b487172242"
server_key_iv: "99cb04ba3871a26491727be13795bdcc"
server_key_version: "encryptionkey@0"
I20260929 19:04:37.228348 19518 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260929 19:04:37.229077 19533 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:04:37.229239 19518 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260929 19:04:37.229336 19518 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708447806542-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708447806542-392-0/minicluster-data/master-0/wal
uuid: "3d12b61530af4068bc771aa9c1ad7316"
format_stamp: "Formatted at 2026-09-29 19:04:37 on dist-test-slave-lqkv"
server_key: "f81da2a074bc060c025e60b487172242"
server_key_iv: "99cb04ba3871a26491727be13795bdcc"
server_key_version: "encryptionkey@0"
I20260929 19:04:37.229413 19518 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708447806542-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708447806542-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708447806542-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:04:37.237663 19518 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:04:37.239658 19518 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:04:37.239806 19518 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:04:37.243531 19518 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43741
I20260929 19:04:37.243557 19585 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43741 every 8 connection(s)
I20260929 19:04:37.243845 19518 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708447806542-392-0/minicluster-data/master-0/data/info.pb
I20260929 19:04:37.244585 19586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:04:37.246611 19586 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d12b61530af4068bc771aa9c1ad7316: Bootstrap starting.
I20260929 19:04:37.247134 19586 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3d12b61530af4068bc771aa9c1ad7316: Neither blocks nor log segments found. Creating new log.
I20260929 19:04:37.247416 19586 log.cc:824] T 00000000000000000000000000000000 P 3d12b61530af4068bc771aa9c1ad7316: Log is configured to *not* fsync() on all Append() calls
I20260929 19:04:37.248049 19586 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d12b61530af4068bc771aa9c1ad7316: No bootstrap required, opened a new log
I20260929 19:04:37.249562 19586 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3d12b61530af4068bc771aa9c1ad7316 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d12b61530af4068bc771aa9c1ad7316" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43741 } }
I20260929 19:04:37.249677 19586 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3d12b61530af4068bc771aa9c1ad7316 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:04:37.249698 19586 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3d12b61530af4068bc771aa9c1ad7316 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d12b61530af4068bc771aa9c1ad7316, State: Initialized, Role: FOLLOWER
I20260929 19:04:37.249851 19586 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3d12b61530af4068bc771aa9c1ad7316 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d12b61530af4068bc771aa9c1ad7316" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43741 } }
I20260929 19:04:37.249914 19586 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3d12b61530af4068bc771aa9c1ad7316 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 19:04:37.249953 19586 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3d12b61530af4068bc771aa9c1ad7316 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 19:04:37.250001 19586 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3d12b61530af4068bc771aa9c1ad7316 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:04:37.250460 19586 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3d12b61530af4068bc771aa9c1ad7316 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d12b61530af4068bc771aa9c1ad7316" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43741 } }
I20260929 19:04:37.250542 19586 leader_election.cc:302] T 00000000000000000000000000000000 P 3d12b61530af4068bc771aa9c1ad7316 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3d12b61530af4068bc771aa9c1ad7316; no voters:
I20260929 19:04:37.250661 19586 leader_election.cc:288] T 00000000000000000000000000000000 P 3d12b61530af4068bc771aa9c1ad7316 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 19:04:37.250739 19589 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3d12b61530af4068bc771aa9c1ad7316 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:04:37.250886 19586 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3d12b61530af4068bc771aa9c1ad7316 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 19:04:37.250908 19589 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3d12b61530af4068bc771aa9c1ad7316 [term 1 LEADER]: Becoming Leader. State: Replica: 3d12b61530af4068bc771aa9c1ad7316, State: Running, Role: LEADER
I20260929 19:04:37.251068 19589 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3d12b61530af4068bc771aa9c1ad7316 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d12b61530af4068bc771aa9c1ad7316" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43741 } }
I20260929 19:04:37.251529 19590 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d12b61530af4068bc771aa9c1ad7316 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3d12b61530af4068bc771aa9c1ad7316" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d12b61530af4068bc771aa9c1ad7316" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43741 } } }
I20260929 19:04:37.251614 19590 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d12b61530af4068bc771aa9c1ad7316 [sys.catalog]: This master's current role is: LEADER
I20260929 19:04:37.251807 19591 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d12b61530af4068bc771aa9c1ad7316 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3d12b61530af4068bc771aa9c1ad7316. Latest consensus state: current_term: 1 leader_uuid: "3d12b61530af4068bc771aa9c1ad7316" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d12b61530af4068bc771aa9c1ad7316" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43741 } } }
I20260929 19:04:37.251860 19598 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 19:04:37.251894 19591 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d12b61530af4068bc771aa9c1ad7316 [sys.catalog]: This master's current role is: LEADER
I20260929 19:04:37.252391 19598 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 19:04:37.253198 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUbnLHE/build/release/bin/kudu as pid 19518
I20260929 19:04:37.253341 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708447806542-392-0/minicluster-data/master-0/wal/instance
I20260929 19:04:37.253575 392 external_mini_cluster.cc:1428] Setting key d237888a5e962c2628744a9ead3d0868
I20260929 19:04:37.254047 19598 catalog_manager.cc:1347] Generated new cluster ID: 6e313d0348e7481d9f73408bd1bf5896
I20260929 19:04:37.254096 19598 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 29 19:04:37 dist-test-slave-lqkv krb5kdc[19505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708677, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 19:04:37.267131 19609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:37.254159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33288 (local address 127.0.98.62:43741)
0929 19:04:37.256976 (+ 2817us) server_negotiation.cc:206] Beginning negotiation
0929 19:04:37.256984 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 19:04:37.257011 (+ 27us) server_negotiation.cc:407] Connection header received
0929 19:04:37.257109 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:04:37.257115 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:04:37.257163 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:04:37.257280 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:04:37.258039 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:37.259334 (+ 1295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:37.260413 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:37.260979 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:37.261069 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:04:37.263691 (+ 2622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:04:37.263696 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:04:37.263696 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:04:37.263707 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:04:37.265590 (+ 1883us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:04:37.265968 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:04:37.265971 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:04:37.265971 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:04:37.266071 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:04:37.266265 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:04:37.266266 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:04:37.266266 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:04:37.266566 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:04:37.266692 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0929 19:04:37.266794 (+ 102us) server_negotiation.cc:299] Negotiation successful
0929 19:04:37.266920 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2786,"thread_start_us":81,"threads_started":1}
I20260929 19:04:37.270534 19598 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 19:04:37.270990 19598 catalog_manager.cc:1504] Loading token signing keys...
I20260929 19:04:37.279201 19598 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3d12b61530af4068bc771aa9c1ad7316: Generated new TSK 0
I20260929 19:04:37.279345 19598 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708447806542-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708447806542-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260929 19:04:37.309206 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUbnLHE/build/release/bin/kudu
/tmp/dist-test-taskUbnLHE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708447806542-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708447806542-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708447806542-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708447806542-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43741
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708447806542-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708447806542-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708447806542-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708447806542-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:04:37.396657 19612 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:04:37.396857 19612 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:04:37.396879 19612 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:04:37.398008 19612 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:04:37.398044 19612 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:04:37.398090 19612 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 19:04:37.399398 19612 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708447806542-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708447806542-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708447806542-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708447806542-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43741
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19612
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708447806542-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 18:09:13 UTC on ccycloud.quasar-hpkalf.root.comops.site
build id 4680913
I20260929 19:04:37.399637 19612 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:04:37.399847 19612 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:04:37.404531 19618 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:04:37.404536 19617 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:04:37.404652 19612 server_base.cc:1034] running on GCE node
W20260929 19:04:37.404731 19620 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:04:37.405017 19612 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:04:37.405236 19612 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:04:37.406394 19612 hybrid_clock.cc:648] HybridClock initialized: now 1790708677406387 us; error 35 us; skew 500 ppm
Sep 29 19:04:37 dist-test-slave-lqkv krb5kdc[19505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708677, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:04:37.408639 19612 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260929 19:04:37.409060 19612 webserver.cc:460] Webserver started at http://127.0.98.1:46513/ using document root <none> and password file <none>
I20260929 19:04:37.409263 19612 fs_manager.cc:359] Metadata directory not provided
I20260929 19:04:37.409325 19612 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:04:37.409471 19612 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:04:37.410405 19612 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708447806542-392-0/minicluster-data/ts-0/data/instance:
uuid: "f8dfe7a229a3434abfa32f67e62e36f0"
format_stamp: "Formatted at 2026-09-29 19:04:37 on dist-test-slave-lqkv"
server_key: "d0d35dbdc49c6ad5652df8e41b3fb764"
server_key_iv: "8312d23ea2c10b491a2c66bf32c69e62"
server_key_version: "encryptionkey@0"
I20260929 19:04:37.410739 19612 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708447806542-392-0/minicluster-data/ts-0/wal/instance:
uuid: "f8dfe7a229a3434abfa32f67e62e36f0"
format_stamp: "Formatted at 2026-09-29 19:04:37 on dist-test-slave-lqkv"
server_key: "d0d35dbdc49c6ad5652df8e41b3fb764"
server_key_iv: "8312d23ea2c10b491a2c66bf32c69e62"
server_key_version: "encryptionkey@0"
I20260929 19:04:37.411908 19612 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 19:04:37.412520 19627 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:04:37.412685 19612 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 19:04:37.412775 19612 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708447806542-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708447806542-392-0/minicluster-data/ts-0/wal
uuid: "f8dfe7a229a3434abfa32f67e62e36f0"
format_stamp: "Formatted at 2026-09-29 19:04:37 on dist-test-slave-lqkv"
server_key: "d0d35dbdc49c6ad5652df8e41b3fb764"
server_key_iv: "8312d23ea2c10b491a2c66bf32c69e62"
server_key_version: "encryptionkey@0"
I20260929 19:04:37.412855 19612 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708447806542-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708447806542-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708447806542-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:04:37.421602 19612 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:04:37.423187 19612 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:04:37.423328 19612 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:04:37.423528 19612 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:04:37.423779 19612 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:04:37.423827 19612 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:04:37.423870 19612 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:04:37.423897 19612 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:04:37.429572 19612 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43215
I20260929 19:04:37.429651 19740 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43215 every 8 connection(s)
I20260929 19:04:37.429921 19612 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708447806542-392-0/minicluster-data/ts-0/data/info.pb
I20260929 19:04:37.433631 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUbnLHE/build/release/bin/kudu as pid 19612
I20260929 19:04:37.433753 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708447806542-392-0/minicluster-data/ts-0/wal/instance
I20260929 19:04:37.433974 392 external_mini_cluster.cc:1428] Setting key faf97797eeb640ff4f07d2ce31159d4e
Sep 29 19:04:37 dist-test-slave-lqkv krb5kdc[19505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708677, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 19:04:37.439075 19609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:37.430535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44167 (local address 127.0.98.62:43741)
0929 19:04:37.430574 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0929 19:04:37.430575 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 19:04:37.430821 (+ 246us) server_negotiation.cc:407] Connection header received
0929 19:04:37.431694 (+ 873us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:04:37.431695 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:04:37.431710 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:04:37.431771 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:04:37.432329 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:37.432911 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:37.433945 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:37.434152 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:37.434232 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:04:37.436421 (+ 2189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:04:37.436422 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:04:37.436422 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:04:37.436428 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:04:37.437833 (+ 1405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:04:37.438210 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:04:37.438211 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:04:37.438212 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:04:37.438268 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:04:37.438504 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:04:37.438505 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:04:37.438506 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:04:37.438606 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:04:37.438669 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0929 19:04:37.438915 (+ 246us) server_negotiation.cc:299] Negotiation successful
0929 19:04:37.438993 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260929 19:04:37.439260 19743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:37.430524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43741 (local address 127.0.98.1:44167)
0929 19:04:37.430700 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0929 19:04:37.430731 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0929 19:04:37.431590 (+ 859us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:04:37.431831 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:04:37.431835 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:04:37.432066 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:04:37.432226 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:04:37.432230 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:04:37.432997 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:04:37.432998 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:04:37.433815 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:04:37.433817 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:04:37.434242 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:04:37.434243 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:04:37.434379 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:04:37.434743 (+ 364us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:04:37.434750 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:04:37.436328 (+ 1578us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:04:37.437927 (+ 1599us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:04:37.437934 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:04:37.437935 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:04:37.438131 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:04:37.438328 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:04:37.438328 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:04:37.438329 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:04:37.438436 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:04:37.438675 (+ 239us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:04:37.438681 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:04:37.438800 (+ 119us) client_negotiation.cc:769] Sending connection context
0929 19:04:37.438893 (+ 93us) client_negotiation.cc:240] Negotiation successful
0929 19:04:37.439022 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":77,"threads_started":1}
I20260929 19:04:37.439664 19741 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43741
I20260929 19:04:37.439755 19741 heartbeater.cc:464] Registering TS with master...
I20260929 19:04:37.439918 19741 heartbeater.cc:511] Master 127.0.98.62:43741 requested a full tablet report, sending...
I20260929 19:04:37.440358 19550 ts_manager.cc:194] Registered new tserver with Master: f8dfe7a229a3434abfa32f67e62e36f0 (127.0.98.1:43215)
I20260929 19:04:37.440990 19550 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:44167
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708447806542-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708447806542-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260929 19:04:37.457098 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUbnLHE/build/release/bin/kudu
/tmp/dist-test-taskUbnLHE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708447806542-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708447806542-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708447806542-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708447806542-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43741
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708447806542-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708447806542-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708447806542-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708447806542-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:04:37.530719 19746 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:04:37.530889 19746 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:04:37.530910 19746 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:04:37.531999 19746 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:04:37.532044 19746 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:04:37.532068 19746 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 19:04:37.533402 19746 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708447806542-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708447806542-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708447806542-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708447806542-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43741
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19746
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708447806542-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 18:09:13 UTC on ccycloud.quasar-hpkalf.root.comops.site
build id 4680913
I20260929 19:04:37.533547 19746 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:04:37.533684 19746 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:04:37.538033 19751 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:04:37.538046 19754 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:04:37.538038 19752 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:04:37.538224 19746 server_base.cc:1034] running on GCE node
I20260929 19:04:37.538460 19746 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:04:37.538663 19746 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:04:37.539822 19746 hybrid_clock.cc:648] HybridClock initialized: now 1790708677539820 us; error 37 us; skew 500 ppm
Sep 29 19:04:37 dist-test-slave-lqkv krb5kdc[19505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708677, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:04:37.542155 19746 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260929 19:04:37.542613 19746 webserver.cc:460] Webserver started at http://127.0.98.2:39929/ using document root <none> and password file <none>
I20260929 19:04:37.542836 19746 fs_manager.cc:359] Metadata directory not provided
I20260929 19:04:37.542905 19746 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:04:37.543043 19746 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:04:37.543979 19746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708447806542-392-0/minicluster-data/ts-1/data/instance:
uuid: "a743c93896d1453e9cc61bf7d69ae333"
format_stamp: "Formatted at 2026-09-29 19:04:37 on dist-test-slave-lqkv"
server_key: "1fa8efee62825d8150eb0ce6fede8abb"
server_key_iv: "4b973f19ba7b4fa7e44201d21c6ecaff"
server_key_version: "encryptionkey@0"
I20260929 19:04:37.544304 19746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708447806542-392-0/minicluster-data/ts-1/wal/instance:
uuid: "a743c93896d1453e9cc61bf7d69ae333"
format_stamp: "Formatted at 2026-09-29 19:04:37 on dist-test-slave-lqkv"
server_key: "1fa8efee62825d8150eb0ce6fede8abb"
server_key_iv: "4b973f19ba7b4fa7e44201d21c6ecaff"
server_key_version: "encryptionkey@0"
I20260929 19:04:37.545611 19746 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 19:04:37.546278 19761 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:04:37.546428 19746 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 19:04:37.546514 19746 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708447806542-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708447806542-392-0/minicluster-data/ts-1/wal
uuid: "a743c93896d1453e9cc61bf7d69ae333"
format_stamp: "Formatted at 2026-09-29 19:04:37 on dist-test-slave-lqkv"
server_key: "1fa8efee62825d8150eb0ce6fede8abb"
server_key_iv: "4b973f19ba7b4fa7e44201d21c6ecaff"
server_key_version: "encryptionkey@0"
I20260929 19:04:37.546591 19746 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708447806542-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708447806542-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708447806542-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:04:37.589221 19746 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:04:37.591188 19746 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:04:37.591327 19746 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:04:37.591558 19746 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:04:37.591852 19746 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:04:37.591889 19746 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:04:37.591943 19746 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:04:37.591969 19746 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:04:37.597561 19746 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43069
I20260929 19:04:37.597602 19874 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43069 every 8 connection(s)
I20260929 19:04:37.597888 19746 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708447806542-392-0/minicluster-data/ts-1/data/info.pb
Sep 29 19:04:37 dist-test-slave-lqkv krb5kdc[19505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708677, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 19:04:37.604681 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUbnLHE/build/release/bin/kudu as pid 19746
I20260929 19:04:37.604787 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708447806542-392-0/minicluster-data/ts-1/wal/instance
I20260929 19:04:37.604990 392 external_mini_cluster.cc:1428] Setting key 3582c5c448a877ab7ac126ccd4f4a091
I20260929 19:04:37.606568 19609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:37.598536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51427 (local address 127.0.98.62:43741)
0929 19:04:37.598576 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0929 19:04:37.598578 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:04:37.598745 (+ 167us) server_negotiation.cc:407] Connection header received
0929 19:04:37.599514 (+ 769us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:04:37.599515 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:04:37.599522 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:04:37.599572 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:04:37.600090 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:37.600532 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:37.601543 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:37.601748 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:37.601810 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:04:37.603552 (+ 1742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:04:37.603553 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:04:37.603553 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:04:37.603558 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:04:37.604939 (+ 1381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:04:37.605395 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:04:37.605396 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:04:37.605397 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:04:37.605467 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:04:37.605898 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:04:37.605899 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:04:37.605899 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:04:37.606040 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:04:37.606112 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 19:04:37.606446 (+ 334us) server_negotiation.cc:299] Negotiation successful
0929 19:04:37.606508 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260929 19:04:37.606846 19877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:37.598479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43741 (local address 127.0.98.2:51427)
0929 19:04:37.598625 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0929 19:04:37.598661 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0929 19:04:37.599436 (+ 775us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:04:37.599637 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:04:37.599639 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:04:37.599854 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:04:37.600024 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:04:37.600028 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:04:37.600600 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:04:37.600601 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:04:37.601462 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:04:37.601464 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:04:37.601825 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:04:37.601826 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:04:37.601924 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:04:37.602304 (+ 380us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:04:37.602311 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:04:37.603465 (+ 1154us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:04:37.605003 (+ 1538us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:04:37.605009 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:04:37.605011 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:04:37.605276 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:04:37.605532 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:04:37.605532 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:04:37.605533 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:04:37.605656 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:04:37.606144 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:04:37.606150 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:04:37.606312 (+ 162us) client_negotiation.cc:769] Sending connection context
0929 19:04:37.606424 (+ 112us) client_negotiation.cc:240] Negotiation successful
0929 19:04:37.606588 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":91,"threads_started":1}
I20260929 19:04:37.607369 19875 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43741
I20260929 19:04:37.607474 19875 heartbeater.cc:464] Registering TS with master...
I20260929 19:04:37.607633 19875 heartbeater.cc:511] Master 127.0.98.62:43741 requested a full tablet report, sending...
I20260929 19:04:37.608006 19550 ts_manager.cc:194] Registered new tserver with Master: a743c93896d1453e9cc61bf7d69ae333 (127.0.98.2:43069)
I20260929 19:04:37.608374 19550 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:51427
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708447806542-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708447806542-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260929 19:04:37.628624 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUbnLHE/build/release/bin/kudu
/tmp/dist-test-taskUbnLHE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708447806542-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708447806542-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708447806542-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708447806542-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43741
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708447806542-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708447806542-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708447806542-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708447806542-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:04:37.705662 19880 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:04:37.705855 19880 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:04:37.705890 19880 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:04:37.706983 19880 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:04:37.707047 19880 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:04:37.707093 19880 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 19:04:37.708540 19880 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708447806542-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708447806542-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708447806542-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708447806542-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43741
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19880
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708447806542-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 18:09:13 UTC on ccycloud.quasar-hpkalf.root.comops.site
build id 4680913
I20260929 19:04:37.708771 19880 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:04:37.708987 19880 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:04:37.713459 19886 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:04:37.713457 19885 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:04:37.713465 19888 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:04:37.713620 19880 server_base.cc:1034] running on GCE node
I20260929 19:04:37.713778 19880 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:04:37.713937 19880 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:04:37.715070 19880 hybrid_clock.cc:648] HybridClock initialized: now 1790708677715052 us; error 32 us; skew 500 ppm
Sep 29 19:04:37 dist-test-slave-lqkv krb5kdc[19505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708677, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:04:37.717372 19880 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260929 19:04:37.717828 19880 webserver.cc:460] Webserver started at http://127.0.98.3:37423/ using document root <none> and password file <none>
I20260929 19:04:37.718082 19880 fs_manager.cc:359] Metadata directory not provided
I20260929 19:04:37.718127 19880 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:04:37.718261 19880 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:04:37.719187 19880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708447806542-392-0/minicluster-data/ts-2/data/instance:
uuid: "6746ef368157479ab9667559078d67f7"
format_stamp: "Formatted at 2026-09-29 19:04:37 on dist-test-slave-lqkv"
server_key: "1da8f7c9bb948f67078b530093e4487d"
server_key_iv: "5e4d91fb64f15311615f88463c57649f"
server_key_version: "encryptionkey@0"
I20260929 19:04:37.719547 19880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708447806542-392-0/minicluster-data/ts-2/wal/instance:
uuid: "6746ef368157479ab9667559078d67f7"
format_stamp: "Formatted at 2026-09-29 19:04:37 on dist-test-slave-lqkv"
server_key: "1da8f7c9bb948f67078b530093e4487d"
server_key_iv: "5e4d91fb64f15311615f88463c57649f"
server_key_version: "encryptionkey@0"
I20260929 19:04:37.720732 19880 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260929 19:04:37.721422 19895 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:04:37.721587 19880 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 19:04:37.721650 19880 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708447806542-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708447806542-392-0/minicluster-data/ts-2/wal
uuid: "6746ef368157479ab9667559078d67f7"
format_stamp: "Formatted at 2026-09-29 19:04:37 on dist-test-slave-lqkv"
server_key: "1da8f7c9bb948f67078b530093e4487d"
server_key_iv: "5e4d91fb64f15311615f88463c57649f"
server_key_version: "encryptionkey@0"
I20260929 19:04:37.721732 19880 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708447806542-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708447806542-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708447806542-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:04:37.742369 19880 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:04:37.744171 19880 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:04:37.744323 19880 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:04:37.744544 19880 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:04:37.744854 19880 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:04:37.744899 19880 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:04:37.744935 19880 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:04:37.744961 19880 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:04:37.750810 19880 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46703
I20260929 19:04:37.750849 20008 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46703 every 8 connection(s)
I20260929 19:04:37.751163 19880 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708447806542-392-0/minicluster-data/ts-2/data/info.pb
I20260929 19:04:37.752779 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUbnLHE/build/release/bin/kudu as pid 19880
I20260929 19:04:37.752921 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708447806542-392-0/minicluster-data/ts-2/wal/instance
I20260929 19:04:37.753101 392 external_mini_cluster.cc:1428] Setting key 3782dde391bea54d2da1792ab9ce6257
Sep 29 19:04:37 dist-test-slave-lqkv krb5kdc[19505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708677, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 19:04:37.759830 19609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:37.751730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55877 (local address 127.0.98.62:43741)
0929 19:04:37.751766 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0929 19:04:37.751768 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:04:37.751968 (+ 200us) server_negotiation.cc:407] Connection header received
0929 19:04:37.753212 (+ 1244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:04:37.753213 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:04:37.753224 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:04:37.753275 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:04:37.754088 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:37.754551 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:37.755444 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:37.755611 (+ 167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:37.755658 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:04:37.757744 (+ 2086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:04:37.757745 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:04:37.757745 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:04:37.757750 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:04:37.758823 (+ 1073us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:04:37.759158 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:04:37.759159 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:04:37.759159 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:04:37.759202 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:04:37.759394 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:04:37.759395 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:04:37.759395 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:04:37.759481 (+ 86us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:04:37.759526 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0929 19:04:37.759723 (+ 197us) server_negotiation.cc:299] Negotiation successful
0929 19:04:37.759774 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260929 19:04:37.760020 20011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:37.751722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43741 (local address 127.0.98.3:55877)
0929 19:04:37.751844 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0929 19:04:37.751882 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 19:04:37.753073 (+ 1191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:04:37.753422 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:04:37.753426 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:04:37.753748 (+ 322us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:04:37.754007 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:04:37.754012 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:04:37.754629 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:04:37.754630 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:04:37.755376 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:04:37.755378 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:04:37.755675 (+ 297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:04:37.755675 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:04:37.755767 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:04:37.756239 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:04:37.756247 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:04:37.757654 (+ 1407us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:04:37.758893 (+ 1239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:04:37.758898 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:04:37.758899 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:04:37.759098 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:04:37.759254 (+ 156us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:04:37.759254 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:04:37.759255 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:04:37.759338 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:04:37.759529 (+ 191us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:04:37.759533 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:04:37.759626 (+ 93us) client_negotiation.cc:769] Sending connection context
0929 19:04:37.759698 (+ 72us) client_negotiation.cc:240] Negotiation successful
0929 19:04:37.759796 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":60,"threads_started":1}
I20260929 19:04:37.760475 20009 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43741
I20260929 19:04:37.760565 20009 heartbeater.cc:464] Registering TS with master...
I20260929 19:04:37.760717 20009 heartbeater.cc:511] Master 127.0.98.62:43741 requested a full tablet report, sending...
I20260929 19:04:37.761062 19550 ts_manager.cc:194] Registered new tserver with Master: 6746ef368157479ab9667559078d67f7 (127.0.98.3:46703)
I20260929 19:04:37.761361 19550 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:55877
I20260929 19:04:37.764529 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 19:04:37.771232 19609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:37.765780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33292 (local address 127.0.98.62:43741)
0929 19:04:37.765815 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0929 19:04:37.765816 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 19:04:37.765979 (+ 163us) server_negotiation.cc:407] Connection header received
0929 19:04:37.766047 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:04:37.766047 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:04:37.766057 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:04:37.766107 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:04:37.766662 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:37.767096 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:37.767827 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:37.768007 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:37.768085 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:04:37.769166 (+ 1081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:04:37.769168 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:04:37.769168 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:04:37.769174 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:04:37.770184 (+ 1010us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:04:37.770508 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:04:37.770509 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:04:37.770510 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:04:37.770565 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:04:37.770746 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:04:37.770747 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:04:37.770747 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:04:37.770868 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:04:37.770967 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0929 19:04:37.771080 (+ 113us) server_negotiation.cc:299] Negotiation successful
0929 19:04:37.771146 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260929 19:04:37.772661 19550 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33292:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 19:04:37.774173 19550 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 19:04:37.779922 20021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:37.776038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43215 (local address 127.0.0.1:45878)
0929 19:04:37.776366 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0929 19:04:37.776389 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 19:04:37.776456 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:04:37.776698 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:04:37.776701 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:04:37.776703 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:04:37.776759 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:04:37.776761 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:04:37.777792 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:04:37.777793 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:04:37.778881 (+ 1088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:04:37.778884 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:04:37.779647 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:04:37.779648 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:04:37.779730 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:04:37.779742 (+ 12us) client_negotiation.cc:769] Sending connection context
0929 19:04:37.779807 (+ 65us) client_negotiation.cc:240] Negotiation successful
0929 19:04:37.779856 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"mutex_wait_us":4,"spinlock_wait_cycles":768,"thread_start_us":54,"threads_started":1}
I20260929 19:04:37.779942 20020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:37.776151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45878 (local address 127.0.98.1:43215)
0929 19:04:37.776304 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0929 19:04:37.776309 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 19:04:37.776549 (+ 240us) server_negotiation.cc:407] Connection header received
0929 19:04:37.776599 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:04:37.776601 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:04:37.776613 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:04:37.776694 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:04:37.776842 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:37.777694 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:37.778971 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:37.779568 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:37.779648 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:04:37.779702 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0929 19:04:37.779827 (+ 125us) server_negotiation.cc:299] Negotiation successful
0929 19:04:37.779865 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":89,"threads_started":1}
I20260929 19:04:37.780323 20019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:37.776115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46058 (local address 127.0.98.3:46703)
0929 19:04:37.776255 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0929 19:04:37.776259 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 19:04:37.776454 (+ 195us) server_negotiation.cc:407] Connection header received
0929 19:04:37.776699 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:04:37.776702 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:04:37.776718 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:04:37.776765 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:04:37.777017 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:37.778068 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:37.779070 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:37.779692 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:37.779734 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:04:37.779778 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0929 19:04:37.780210 (+ 432us) server_negotiation.cc:299] Negotiation successful
0929 19:04:37.780254 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":76,"threads_started":1}
I20260929 19:04:37.780373 20017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:37.776210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43069 (local address 127.0.0.1:57152)
0929 19:04:37.776552 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0929 19:04:37.776561 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 19:04:37.776605 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:04:37.776873 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:04:37.776874 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:04:37.776877 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:04:37.776939 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:04:37.776940 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:04:37.778184 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:04:37.778185 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:04:37.779245 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:04:37.779247 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:04:37.780140 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:04:37.780141 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:04:37.780201 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:04:37.780204 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 19:04:37.780251 (+ 47us) client_negotiation.cc:240] Negotiation successful
0929 19:04:37.780284 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"mutex_wait_us":33,"thread_start_us":34,"threads_started":1}
I20260929 19:04:37.780516 20022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:37.776367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57152 (local address 127.0.98.2:43069)
0929 19:04:37.776545 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0929 19:04:37.776549 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 19:04:37.776596 (+ 47us) server_negotiation.cc:407] Connection header received
0929 19:04:37.776669 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:04:37.776671 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:04:37.776686 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:04:37.776761 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:04:37.777015 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:37.778087 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:37.779323 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:37.780010 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:37.780065 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:04:37.780111 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0929 19:04:37.780281 (+ 170us) server_negotiation.cc:299] Negotiation successful
0929 19:04:37.780328 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":84,"threads_started":1}
I20260929 19:04:37.780658 20018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:37.776038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46703 (local address 127.0.0.1:46058)
0929 19:04:37.776337 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0929 19:04:37.776375 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 19:04:37.776461 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:04:37.776886 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:04:37.776887 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:04:37.776889 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:04:37.776936 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:04:37.776938 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:04:37.778187 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:04:37.778188 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:04:37.778981 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:04:37.778983 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:04:37.779912 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:04:37.779913 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:04:37.779971 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:04:37.779975 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 19:04:37.780527 (+ 552us) client_negotiation.cc:240] Negotiation successful
0929 19:04:37.780580 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"mutex_wait_us":30,"thread_start_us":55,"threads_started":1}
I20260929 19:04:37.780737 19675 tablet_service.cc:1467] Processing CreateTablet for tablet fbd21249063f43158bb9e08816b25d60 (DEFAULT_TABLE table=test-table [id=fe84676a7162467cb105d5ea9f882aaf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:04:37.781107 19675 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fbd21249063f43158bb9e08816b25d60. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:04:37.781309 19943 tablet_service.cc:1467] Processing CreateTablet for tablet fbd21249063f43158bb9e08816b25d60 (DEFAULT_TABLE table=test-table [id=fe84676a7162467cb105d5ea9f882aaf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:04:37.781571 19943 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fbd21249063f43158bb9e08816b25d60. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:04:37.784373 20023 tablet_bootstrap.cc:492] T fbd21249063f43158bb9e08816b25d60 P f8dfe7a229a3434abfa32f67e62e36f0: Bootstrap starting.
I20260929 19:04:37.784744 20024 tablet_bootstrap.cc:492] T fbd21249063f43158bb9e08816b25d60 P 6746ef368157479ab9667559078d67f7: Bootstrap starting.
I20260929 19:04:37.785163 20023 tablet_bootstrap.cc:654] T fbd21249063f43158bb9e08816b25d60 P f8dfe7a229a3434abfa32f67e62e36f0: Neither blocks nor log segments found. Creating new log.
I20260929 19:04:37.785130 19809 tablet_service.cc:1467] Processing CreateTablet for tablet fbd21249063f43158bb9e08816b25d60 (DEFAULT_TABLE table=test-table [id=fe84676a7162467cb105d5ea9f882aaf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:04:37.785421 19809 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fbd21249063f43158bb9e08816b25d60. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:04:37.785450 20024 tablet_bootstrap.cc:654] T fbd21249063f43158bb9e08816b25d60 P 6746ef368157479ab9667559078d67f7: Neither blocks nor log segments found. Creating new log.
I20260929 19:04:37.785766 20023 log.cc:824] T fbd21249063f43158bb9e08816b25d60 P f8dfe7a229a3434abfa32f67e62e36f0: Log is configured to *not* fsync() on all Append() calls
I20260929 19:04:37.785988 20024 log.cc:824] T fbd21249063f43158bb9e08816b25d60 P 6746ef368157479ab9667559078d67f7: Log is configured to *not* fsync() on all Append() calls
I20260929 19:04:37.786504 20023 tablet_bootstrap.cc:492] T fbd21249063f43158bb9e08816b25d60 P f8dfe7a229a3434abfa32f67e62e36f0: No bootstrap required, opened a new log
I20260929 19:04:37.786584 20023 ts_tablet_manager.cc:1397] T fbd21249063f43158bb9e08816b25d60 P f8dfe7a229a3434abfa32f67e62e36f0: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260929 19:04:37.786700 20024 tablet_bootstrap.cc:492] T fbd21249063f43158bb9e08816b25d60 P 6746ef368157479ab9667559078d67f7: No bootstrap required, opened a new log
I20260929 19:04:37.786772 20024 ts_tablet_manager.cc:1397] T fbd21249063f43158bb9e08816b25d60 P 6746ef368157479ab9667559078d67f7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 19:04:37.788136 20023 raft_consensus.cc:348] T fbd21249063f43158bb9e08816b25d60 P f8dfe7a229a3434abfa32f67e62e36f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8dfe7a229a3434abfa32f67e62e36f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43215 } } peers { permanent_uuid: "a743c93896d1453e9cc61bf7d69ae333" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43069 } } peers { permanent_uuid: "6746ef368157479ab9667559078d67f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46703 } }
I20260929 19:04:37.788285 20023 raft_consensus.cc:374] T fbd21249063f43158bb9e08816b25d60 P f8dfe7a229a3434abfa32f67e62e36f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:04:37.788326 20023 raft_consensus.cc:729] T fbd21249063f43158bb9e08816b25d60 P f8dfe7a229a3434abfa32f67e62e36f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8dfe7a229a3434abfa32f67e62e36f0, State: Initialized, Role: FOLLOWER
I20260929 19:04:37.788452 20023 consensus_queue.cc:260] T fbd21249063f43158bb9e08816b25d60 P f8dfe7a229a3434abfa32f67e62e36f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8dfe7a229a3434abfa32f67e62e36f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43215 } } peers { permanent_uuid: "a743c93896d1453e9cc61bf7d69ae333" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43069 } } peers { permanent_uuid: "6746ef368157479ab9667559078d67f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46703 } }
I20260929 19:04:37.788728 20023 ts_tablet_manager.cc:1428] T fbd21249063f43158bb9e08816b25d60 P f8dfe7a229a3434abfa32f67e62e36f0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 19:04:37.788733 19741 heartbeater.cc:503] Master 127.0.98.62:43741 was elected leader, sending a full tablet report...
I20260929 19:04:37.788949 20027 tablet_bootstrap.cc:492] T fbd21249063f43158bb9e08816b25d60 P a743c93896d1453e9cc61bf7d69ae333: Bootstrap starting.
I20260929 19:04:37.788872 20024 raft_consensus.cc:348] T fbd21249063f43158bb9e08816b25d60 P 6746ef368157479ab9667559078d67f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8dfe7a229a3434abfa32f67e62e36f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43215 } } peers { permanent_uuid: "a743c93896d1453e9cc61bf7d69ae333" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43069 } } peers { permanent_uuid: "6746ef368157479ab9667559078d67f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46703 } }
I20260929 19:04:37.789139 20024 raft_consensus.cc:374] T fbd21249063f43158bb9e08816b25d60 P 6746ef368157479ab9667559078d67f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:04:37.789191 20024 raft_consensus.cc:729] T fbd21249063f43158bb9e08816b25d60 P 6746ef368157479ab9667559078d67f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6746ef368157479ab9667559078d67f7, State: Initialized, Role: FOLLOWER
I20260929 19:04:37.789290 20024 consensus_queue.cc:260] T fbd21249063f43158bb9e08816b25d60 P 6746ef368157479ab9667559078d67f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8dfe7a229a3434abfa32f67e62e36f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43215 } } peers { permanent_uuid: "a743c93896d1453e9cc61bf7d69ae333" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43069 } } peers { permanent_uuid: "6746ef368157479ab9667559078d67f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46703 } }
I20260929 19:04:37.789563 20009 heartbeater.cc:503] Master 127.0.98.62:43741 was elected leader, sending a full tablet report...
I20260929 19:04:37.789584 20027 tablet_bootstrap.cc:654] T fbd21249063f43158bb9e08816b25d60 P a743c93896d1453e9cc61bf7d69ae333: Neither blocks nor log segments found. Creating new log.
I20260929 19:04:37.789565 20024 ts_tablet_manager.cc:1428] T fbd21249063f43158bb9e08816b25d60 P 6746ef368157479ab9667559078d67f7: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 19:04:37.790187 20027 log.cc:824] T fbd21249063f43158bb9e08816b25d60 P a743c93896d1453e9cc61bf7d69ae333: Log is configured to *not* fsync() on all Append() calls
I20260929 19:04:37.790952 20027 tablet_bootstrap.cc:492] T fbd21249063f43158bb9e08816b25d60 P a743c93896d1453e9cc61bf7d69ae333: No bootstrap required, opened a new log
I20260929 19:04:37.791011 20027 ts_tablet_manager.cc:1397] T fbd21249063f43158bb9e08816b25d60 P a743c93896d1453e9cc61bf7d69ae333: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 19:04:37.792271 20027 raft_consensus.cc:348] T fbd21249063f43158bb9e08816b25d60 P a743c93896d1453e9cc61bf7d69ae333 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8dfe7a229a3434abfa32f67e62e36f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43215 } } peers { permanent_uuid: "a743c93896d1453e9cc61bf7d69ae333" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43069 } } peers { permanent_uuid: "6746ef368157479ab9667559078d67f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46703 } }
I20260929 19:04:37.792395 20027 raft_consensus.cc:374] T fbd21249063f43158bb9e08816b25d60 P a743c93896d1453e9cc61bf7d69ae333 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:04:37.792418 20027 raft_consensus.cc:729] T fbd21249063f43158bb9e08816b25d60 P a743c93896d1453e9cc61bf7d69ae333 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a743c93896d1453e9cc61bf7d69ae333, State: Initialized, Role: FOLLOWER
I20260929 19:04:37.792519 20027 consensus_queue.cc:260] T fbd21249063f43158bb9e08816b25d60 P a743c93896d1453e9cc61bf7d69ae333 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8dfe7a229a3434abfa32f67e62e36f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43215 } } peers { permanent_uuid: "a743c93896d1453e9cc61bf7d69ae333" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43069 } } peers { permanent_uuid: "6746ef368157479ab9667559078d67f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46703 } }
I20260929 19:04:37.792776 20027 ts_tablet_manager.cc:1428] T fbd21249063f43158bb9e08816b25d60 P a743c93896d1453e9cc61bf7d69ae333: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 19:04:37.792917 19875 heartbeater.cc:503] Master 127.0.98.62:43741 was elected leader, sending a full tablet report...
I20260929 19:04:37.814596 20029 raft_consensus.cc:482] T fbd21249063f43158bb9e08816b25d60 P 6746ef368157479ab9667559078d67f7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 19:04:37.814685 20029 raft_consensus.cc:504] T fbd21249063f43158bb9e08816b25d60 P 6746ef368157479ab9667559078d67f7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8dfe7a229a3434abfa32f67e62e36f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43215 } } peers { permanent_uuid: "a743c93896d1453e9cc61bf7d69ae333" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43069 } } peers { permanent_uuid: "6746ef368157479ab9667559078d67f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46703 } }
I20260929 19:04:37.814976 20029 leader_election.cc:288] T fbd21249063f43158bb9e08816b25d60 P 6746ef368157479ab9667559078d67f7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f8dfe7a229a3434abfa32f67e62e36f0 (127.0.98.1:43215), a743c93896d1453e9cc61bf7d69ae333 (127.0.98.2:43069)
I20260929 19:04:37.818081 20022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:37.815256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41443 (local address 127.0.98.2:43069)
0929 19:04:37.815310 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0929 19:04:37.815312 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:04:37.815412 (+ 100us) server_negotiation.cc:407] Connection header received
0929 19:04:37.815468 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:04:37.815469 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:04:37.815477 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:04:37.815517 (+ 40us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:04:37.815690 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:37.816323 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:37.817340 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:37.817708 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:37.817774 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:04:37.817801 (+ 27us) server_negotiation.cc:1035] Waiting for connection context
0929 19:04:37.817973 (+ 172us) server_negotiation.cc:299] Negotiation successful
0929 19:04:37.818024 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260929 19:04:37.818094 20032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:37.815189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43069 (local address 127.0.98.3:41443)
0929 19:04:37.815337 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0929 19:04:37.815345 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0929 19:04:37.815406 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:04:37.815530 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:04:37.815531 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:04:37.815532 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:04:37.815586 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:04:37.815587 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:04:37.816401 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:04:37.816401 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:04:37.817260 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:04:37.817262 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:04:37.817807 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:04:37.817808 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:04:37.817878 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:04:37.817881 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 19:04:37.817956 (+ 75us) client_negotiation.cc:240] Negotiation successful
0929 19:04:37.818007 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":97,"threads_started":1}
I20260929 19:04:37.818190 20011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:37.815104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43215 (local address 127.0.98.3:56587)
0929 19:04:37.815139 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0929 19:04:37.815149 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 19:04:37.815192 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:04:37.815455 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:04:37.815456 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:04:37.815457 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:04:37.815516 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:04:37.815517 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:04:37.816389 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:04:37.816390 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:04:37.817279 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:04:37.817283 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:04:37.817838 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:04:37.817839 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:04:37.817951 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:04:37.817955 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 19:04:37.818016 (+ 61us) client_negotiation.cc:240] Negotiation successful
0929 19:04:37.818050 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19,"mutex_wait_us":84}
I20260929 19:04:37.818243 20020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:37.815273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56587 (local address 127.0.98.1:43215)
0929 19:04:37.815330 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0929 19:04:37.815332 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:04:37.815339 (+ 7us) server_negotiation.cc:407] Connection header received
0929 19:04:37.815358 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:04:37.815358 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:04:37.815368 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:04:37.815439 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:04:37.815578 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:37.816287 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:37.817388 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:37.817711 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:37.817830 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:04:37.817838 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0929 19:04:37.818122 (+ 284us) server_negotiation.cc:299] Negotiation successful
0929 19:04:37.818167 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260929 19:04:37.818437 19829 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbd21249063f43158bb9e08816b25d60" candidate_uuid: "6746ef368157479ab9667559078d67f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a743c93896d1453e9cc61bf7d69ae333" is_pre_election: true
I20260929 19:04:37.818548 19829 raft_consensus.cc:2454] T fbd21249063f43158bb9e08816b25d60 P a743c93896d1453e9cc61bf7d69ae333 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6746ef368157479ab9667559078d67f7 in term 0.
I20260929 19:04:37.818531 19695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbd21249063f43158bb9e08816b25d60" candidate_uuid: "6746ef368157479ab9667559078d67f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8dfe7a229a3434abfa32f67e62e36f0" is_pre_election: true
I20260929 19:04:37.818645 19695 raft_consensus.cc:2454] T fbd21249063f43158bb9e08816b25d60 P f8dfe7a229a3434abfa32f67e62e36f0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6746ef368157479ab9667559078d67f7 in term 0.
I20260929 19:04:37.818743 19899 leader_election.cc:302] T fbd21249063f43158bb9e08816b25d60 P 6746ef368157479ab9667559078d67f7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6746ef368157479ab9667559078d67f7, a743c93896d1453e9cc61bf7d69ae333; no voters:
I20260929 19:04:37.818873 20029 raft_consensus.cc:2781] T fbd21249063f43158bb9e08816b25d60 P 6746ef368157479ab9667559078d67f7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 19:04:37.818938 20029 raft_consensus.cc:482] T fbd21249063f43158bb9e08816b25d60 P 6746ef368157479ab9667559078d67f7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 19:04:37.818955 20029 raft_consensus.cc:3037] T fbd21249063f43158bb9e08816b25d60 P 6746ef368157479ab9667559078d67f7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:04:37.819619 20029 raft_consensus.cc:504] T fbd21249063f43158bb9e08816b25d60 P 6746ef368157479ab9667559078d67f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8dfe7a229a3434abfa32f67e62e36f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43215 } } peers { permanent_uuid: "a743c93896d1453e9cc61bf7d69ae333" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43069 } } peers { permanent_uuid: "6746ef368157479ab9667559078d67f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46703 } }
I20260929 19:04:37.819733 20029 leader_election.cc:288] T fbd21249063f43158bb9e08816b25d60 P 6746ef368157479ab9667559078d67f7 [CANDIDATE]: Term 1 election: Requested vote from peers f8dfe7a229a3434abfa32f67e62e36f0 (127.0.98.1:43215), a743c93896d1453e9cc61bf7d69ae333 (127.0.98.2:43069)
I20260929 19:04:37.820015 19829 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbd21249063f43158bb9e08816b25d60" candidate_uuid: "6746ef368157479ab9667559078d67f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a743c93896d1453e9cc61bf7d69ae333"
I20260929 19:04:37.820014 19695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbd21249063f43158bb9e08816b25d60" candidate_uuid: "6746ef368157479ab9667559078d67f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8dfe7a229a3434abfa32f67e62e36f0"
I20260929 19:04:37.820096 19829 raft_consensus.cc:3037] T fbd21249063f43158bb9e08816b25d60 P a743c93896d1453e9cc61bf7d69ae333 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:04:37.820103 19695 raft_consensus.cc:3037] T fbd21249063f43158bb9e08816b25d60 P f8dfe7a229a3434abfa32f67e62e36f0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:04:37.820978 19829 raft_consensus.cc:2454] T fbd21249063f43158bb9e08816b25d60 P a743c93896d1453e9cc61bf7d69ae333 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6746ef368157479ab9667559078d67f7 in term 1.
I20260929 19:04:37.821033 19695 raft_consensus.cc:2454] T fbd21249063f43158bb9e08816b25d60 P f8dfe7a229a3434abfa32f67e62e36f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6746ef368157479ab9667559078d67f7 in term 1.
I20260929 19:04:37.821189 19899 leader_election.cc:302] T fbd21249063f43158bb9e08816b25d60 P 6746ef368157479ab9667559078d67f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6746ef368157479ab9667559078d67f7, f8dfe7a229a3434abfa32f67e62e36f0; no voters:
I20260929 19:04:37.821326 20029 raft_consensus.cc:2781] T fbd21249063f43158bb9e08816b25d60 P 6746ef368157479ab9667559078d67f7 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:04:37.821506 20029 raft_consensus.cc:686] T fbd21249063f43158bb9e08816b25d60 P 6746ef368157479ab9667559078d67f7 [term 1 LEADER]: Becoming Leader. State: Replica: 6746ef368157479ab9667559078d67f7, State: Running, Role: LEADER
I20260929 19:04:37.821599 20029 consensus_queue.cc:237] T fbd21249063f43158bb9e08816b25d60 P 6746ef368157479ab9667559078d67f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8dfe7a229a3434abfa32f67e62e36f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43215 } } peers { permanent_uuid: "a743c93896d1453e9cc61bf7d69ae333" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43069 } } peers { permanent_uuid: "6746ef368157479ab9667559078d67f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46703 } }
I20260929 19:04:37.822381 19549 catalog_manager.cc:5495] T fbd21249063f43158bb9e08816b25d60 P 6746ef368157479ab9667559078d67f7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6746ef368157479ab9667559078d67f7 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "6746ef368157479ab9667559078d67f7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8dfe7a229a3434abfa32f67e62e36f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43215 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a743c93896d1453e9cc61bf7d69ae333" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43069 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6746ef368157479ab9667559078d67f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46703 } health_report { overall_health: HEALTHY } } }
I20260929 19:04:37.828914 20019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:37.826401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46072 (local address 127.0.98.3:46703)
0929 19:04:37.826474 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0929 19:04:37.826476 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:04:37.826484 (+ 8us) server_negotiation.cc:407] Connection header received
0929 19:04:37.826512 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:04:37.826513 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:04:37.826528 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:04:37.826594 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 19:04:37.826684 (+ 90us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:37.827335 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:37.828299 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:37.828467 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:37.828539 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:04:37.828629 (+ 90us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 19:04:37.828702 (+ 73us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 19:04:37.828753 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0929 19:04:37.828782 (+ 29us) server_negotiation.cc:299] Negotiation successful
0929 19:04:37.828796 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 19:04:37.830161 19829 raft_consensus.cc:1261] T fbd21249063f43158bb9e08816b25d60 P a743c93896d1453e9cc61bf7d69ae333 [term 1 FOLLOWER]: Refusing update from remote peer 6746ef368157479ab9667559078d67f7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 19:04:37.830188 19695 raft_consensus.cc:1261] T fbd21249063f43158bb9e08816b25d60 P f8dfe7a229a3434abfa32f67e62e36f0 [term 1 FOLLOWER]: Refusing update from remote peer 6746ef368157479ab9667559078d67f7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 19:04:37.830438 20029 consensus_queue.cc:1035] T fbd21249063f43158bb9e08816b25d60 P 6746ef368157479ab9667559078d67f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a743c93896d1453e9cc61bf7d69ae333" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 19:04:37.830566 20033 consensus_queue.cc:1035] T fbd21249063f43158bb9e08816b25d60 P 6746ef368157479ab9667559078d67f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8dfe7a229a3434abfa32f67e62e36f0" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 19:04:37.832561 20036 mvcc.cc:205] Tried to move back new op lower bound from 7334742744390492160 to 7334742744357941248. Current Snapshot: MvccSnapshot[applied={T|T < 7334742744390492160}]
I20260929 19:04:37.834192 20037 mvcc.cc:205] Tried to move back new op lower bound from 7334742744390492160 to 7334742744357941248. Current Snapshot: MvccSnapshot[applied={T|T < 7334742744390492160 or (T in {7334742744390492160})}]
I20260929 19:04:37.834416 19550 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33292:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 19:04:37.834492 19550 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33292:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 19:04:37.835111 19550 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3d12b61530af4068bc771aa9c1ad7316: Sending DeleteTablet for 3 replicas of tablet fbd21249063f43158bb9e08816b25d60
I20260929 19:04:37.835371 19943 tablet_service.cc:1514] Processing DeleteTablet for tablet fbd21249063f43158bb9e08816b25d60 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:04:37 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46058
I20260929 19:04:37.835381 19675 tablet_service.cc:1514] Processing DeleteTablet for tablet fbd21249063f43158bb9e08816b25d60 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:04:37 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45878
I20260929 19:04:37.835429 19809 tablet_service.cc:1514] Processing DeleteTablet for tablet fbd21249063f43158bb9e08816b25d60 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:04:37 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57152
I20260929 19:04:37.835537 20046 tablet_replica.cc:330] T fbd21249063f43158bb9e08816b25d60 P 6746ef368157479ab9667559078d67f7: stopping tablet replica
I20260929 19:04:37.835620 20046 raft_consensus.cc:2229] T fbd21249063f43158bb9e08816b25d60 P 6746ef368157479ab9667559078d67f7 [term 1 LEADER]: Raft consensus shutting down.
I20260929 19:04:37.835690 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUbnLHE/build/release/bin/kudu with pid 19612
I20260929 19:04:37.836930 20048 tablet_replica.cc:330] T fbd21249063f43158bb9e08816b25d60 P a743c93896d1453e9cc61bf7d69ae333: stopping tablet replica
I20260929 19:04:37.837069 20048 raft_consensus.cc:2229] T fbd21249063f43158bb9e08816b25d60 P a743c93896d1453e9cc61bf7d69ae333 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 19:04:37.837217 20048 raft_consensus.cc:2258] T fbd21249063f43158bb9e08816b25d60 P a743c93896d1453e9cc61bf7d69ae333 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 19:04:37.837575 20046 raft_consensus.cc:2258] T fbd21249063f43158bb9e08816b25d60 P 6746ef368157479ab9667559078d67f7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 19:04:37.837742 20048 ts_tablet_manager.cc:1905] T fbd21249063f43158bb9e08816b25d60 P a743c93896d1453e9cc61bf7d69ae333: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 19:04:37.838004 20046 ts_tablet_manager.cc:1905] T fbd21249063f43158bb9e08816b25d60 P 6746ef368157479ab9667559078d67f7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 19:04:37.839353 20048 ts_tablet_manager.cc:1918] T fbd21249063f43158bb9e08816b25d60 P a743c93896d1453e9cc61bf7d69ae333: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 19:04:37.839401 20046 ts_tablet_manager.cc:1918] T fbd21249063f43158bb9e08816b25d60 P 6746ef368157479ab9667559078d67f7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 19:04:37.839476 20046 log.cc:1196] T fbd21249063f43158bb9e08816b25d60 P 6746ef368157479ab9667559078d67f7: Deleting WAL directory at /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708447806542-392-0/minicluster-data/ts-2/wal/wals/fbd21249063f43158bb9e08816b25d60
I20260929 19:04:37.839541 20048 log.cc:1196] T fbd21249063f43158bb9e08816b25d60 P a743c93896d1453e9cc61bf7d69ae333: Deleting WAL directory at /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790708447806542-392-0/minicluster-data/ts-1/wal/wals/fbd21249063f43158bb9e08816b25d60
I20260929 19:04:37.839711 20046 ts_tablet_manager.cc:1939] T fbd21249063f43158bb9e08816b25d60 P 6746ef368157479ab9667559078d67f7: Deleting consensus metadata
I20260929 19:04:37.839897 20048 ts_tablet_manager.cc:1939] T fbd21249063f43158bb9e08816b25d60 P a743c93896d1453e9cc61bf7d69ae333: Deleting consensus metadata
I20260929 19:04:37.840034 19534 catalog_manager.cc:4841] TS 6746ef368157479ab9667559078d67f7 (127.0.98.3:46703): tablet fbd21249063f43158bb9e08816b25d60 (table test-table [id=fe84676a7162467cb105d5ea9f882aaf]) successfully deleted
I20260929 19:04:37.840312 19537 catalog_manager.cc:4841] TS a743c93896d1453e9cc61bf7d69ae333 (127.0.98.2:43069): tablet fbd21249063f43158bb9e08816b25d60 (table test-table [id=fe84676a7162467cb105d5ea9f882aaf]) successfully deleted
W20260929 19:04:37.841979 19537 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:43215 (error 108)
I20260929 19:04:37.842417 20018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:37.842298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43215 (local address 127.0.0.1:45882)
0929 19:04:37.842352 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0929 19:04:37.842390 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260929 19:04:37.842494 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUbnLHE/build/release/bin/kudu with pid 19746
W20260929 19:04:37.843953 19537 catalog_manager.cc:4584] TS f8dfe7a229a3434abfa32f67e62e36f0 (127.0.98.1:43215): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet fbd21249063f43158bb9e08816b25d60: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43215: connect: Connection refused (error 111)
I20260929 19:04:37.848026 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUbnLHE/build/release/bin/kudu with pid 19880
I20260929 19:04:37.852445 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUbnLHE/build/release/bin/kudu with pid 19518
2026-09-29T19:04:37Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (1787 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-29T19:04:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T19:04:37Z Disabled control of system clock
I20260929 19:04:37.878525 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUbnLHE/build/release/bin/kudu
/tmp/dist-test-taskUbnLHE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708447806542-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708447806542-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708447806542-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708447806542-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44891
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:46157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44891
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 19:04:37.951263 20053 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:04:37.951444 20053 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:04:37.951464 20053 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:04:37.952517 20053 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 19:04:37.952553 20053 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:04:37.952564 20053 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:04:37.952575 20053 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 19:04:37.952585 20053 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 19:04:37.953831 20053 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708447806542-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708447806542-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44891
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44891
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708447806542-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20053
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708447806542-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 18:09:13 UTC on ccycloud.quasar-hpkalf.root.comops.site
build id 4680913
I20260929 19:04:37.954010 20053 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:04:37.954206 20053 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 19:04:37.958712 20053 server_base.cc:1034] running on GCE node
W20260929 19:04:37.958665 20059 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:04:37.958691 20058 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:04:37.958664 20061 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:04:37.959048 20053 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:04:37.959288 20053 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:04:37.960446 20053 hybrid_clock.cc:648] HybridClock initialized: now 1790708677960433 us; error 24 us; skew 500 ppm
I20260929 19:04:37.961608 20053 webserver.cc:460] Webserver started at http://127.0.98.62:37235/ using document root <none> and password file <none>
I20260929 19:04:37.961838 20053 fs_manager.cc:359] Metadata directory not provided
I20260929 19:04:37.961911 20053 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:04:37.962051 20053 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:04:37.962970 20053 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708447806542-392-0/minicluster-data/master-0/data/instance:
uuid: "164ebd7f083c4d4dbc5d04b6bbe83075"
format_stamp: "Formatted at 2026-09-29 19:04:37 on dist-test-slave-lqkv"
server_key: "5c98a0cd66f966d24b0809617ca80bd8"
server_key_iv: "e12014e6e5afe254209a90078993c67f"
server_key_version: "encryptionkey@0"
I20260929 19:04:37.963284 20053 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708447806542-392-0/minicluster-data/master-0/wal/instance:
uuid: "164ebd7f083c4d4dbc5d04b6bbe83075"
format_stamp: "Formatted at 2026-09-29 19:04:37 on dist-test-slave-lqkv"
server_key: "5c98a0cd66f966d24b0809617ca80bd8"
server_key_iv: "e12014e6e5afe254209a90078993c67f"
server_key_version: "encryptionkey@0"
I20260929 19:04:37.964445 20053 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 19:04:37.965093 20067 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:04:37.965234 20053 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260929 19:04:37.965322 20053 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708447806542-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708447806542-392-0/minicluster-data/master-0/wal
uuid: "164ebd7f083c4d4dbc5d04b6bbe83075"
format_stamp: "Formatted at 2026-09-29 19:04:37 on dist-test-slave-lqkv"
server_key: "5c98a0cd66f966d24b0809617ca80bd8"
server_key_iv: "e12014e6e5afe254209a90078993c67f"
server_key_version: "encryptionkey@0"
I20260929 19:04:37.965397 20053 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708447806542-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708447806542-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708447806542-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:04:37.982975 20053 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:04:37.983196 20053 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:04:37.983335 20053 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:04:37.986572 20053 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44891
I20260929 19:04:37.986630 20119 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44891 every 8 connection(s)
I20260929 19:04:37.986919 20053 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708447806542-392-0/minicluster-data/master-0/data/info.pb
I20260929 19:04:37.987751 20120 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:04:37.989750 20120 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 164ebd7f083c4d4dbc5d04b6bbe83075: Bootstrap starting.
I20260929 19:04:37.990253 20120 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 164ebd7f083c4d4dbc5d04b6bbe83075: Neither blocks nor log segments found. Creating new log.
I20260929 19:04:37.990520 20120 log.cc:824] T 00000000000000000000000000000000 P 164ebd7f083c4d4dbc5d04b6bbe83075: Log is configured to *not* fsync() on all Append() calls
I20260929 19:04:37.991112 20120 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 164ebd7f083c4d4dbc5d04b6bbe83075: No bootstrap required, opened a new log
I20260929 19:04:37.992116 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUbnLHE/build/release/bin/kudu as pid 20053
I20260929 19:04:37.992235 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708447806542-392-0/minicluster-data/master-0/wal/instance
I20260929 19:04:37.992427 392 external_mini_cluster.cc:1428] Setting key 76b28ae74cd34cf86122234b568221f2
I20260929 19:04:37.992887 20120 raft_consensus.cc:348] T 00000000000000000000000000000000 P 164ebd7f083c4d4dbc5d04b6bbe83075 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "164ebd7f083c4d4dbc5d04b6bbe83075" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44891 } }
I20260929 19:04:37.993054 20120 raft_consensus.cc:374] T 00000000000000000000000000000000 P 164ebd7f083c4d4dbc5d04b6bbe83075 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:04:37.993096 20120 raft_consensus.cc:729] T 00000000000000000000000000000000 P 164ebd7f083c4d4dbc5d04b6bbe83075 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 164ebd7f083c4d4dbc5d04b6bbe83075, State: Initialized, Role: FOLLOWER
I20260929 19:04:37.993206 20120 consensus_queue.cc:260] T 00000000000000000000000000000000 P 164ebd7f083c4d4dbc5d04b6bbe83075 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "164ebd7f083c4d4dbc5d04b6bbe83075" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44891 } }
I20260929 19:04:37.993270 20120 raft_consensus.cc:388] T 00000000000000000000000000000000 P 164ebd7f083c4d4dbc5d04b6bbe83075 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 19:04:37.993296 20120 raft_consensus.cc:482] T 00000000000000000000000000000000 P 164ebd7f083c4d4dbc5d04b6bbe83075 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 19:04:37.993361 20120 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 164ebd7f083c4d4dbc5d04b6bbe83075 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:04:37.994316 20120 raft_consensus.cc:504] T 00000000000000000000000000000000 P 164ebd7f083c4d4dbc5d04b6bbe83075 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "164ebd7f083c4d4dbc5d04b6bbe83075" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44891 } }
I20260929 19:04:37.994419 20120 leader_election.cc:302] T 00000000000000000000000000000000 P 164ebd7f083c4d4dbc5d04b6bbe83075 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 164ebd7f083c4d4dbc5d04b6bbe83075; no voters:
I20260929 19:04:37.994552 20120 leader_election.cc:288] T 00000000000000000000000000000000 P 164ebd7f083c4d4dbc5d04b6bbe83075 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 19:04:37.994650 20125 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 164ebd7f083c4d4dbc5d04b6bbe83075 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:04:37.994776 20125 raft_consensus.cc:686] T 00000000000000000000000000000000 P 164ebd7f083c4d4dbc5d04b6bbe83075 [term 1 LEADER]: Becoming Leader. State: Replica: 164ebd7f083c4d4dbc5d04b6bbe83075, State: Running, Role: LEADER
I20260929 19:04:37.994800 20120 sys_catalog.cc:564] T 00000000000000000000000000000000 P 164ebd7f083c4d4dbc5d04b6bbe83075 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 19:04:37.994853 20125 consensus_queue.cc:237] T 00000000000000000000000000000000 P 164ebd7f083c4d4dbc5d04b6bbe83075 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "164ebd7f083c4d4dbc5d04b6bbe83075" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44891 } }
I20260929 19:04:37.995142 20125 sys_catalog.cc:455] T 00000000000000000000000000000000 P 164ebd7f083c4d4dbc5d04b6bbe83075 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 164ebd7f083c4d4dbc5d04b6bbe83075. Latest consensus state: current_term: 1 leader_uuid: "164ebd7f083c4d4dbc5d04b6bbe83075" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "164ebd7f083c4d4dbc5d04b6bbe83075" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44891 } } }
I20260929 19:04:37.995211 20125 sys_catalog.cc:458] T 00000000000000000000000000000000 P 164ebd7f083c4d4dbc5d04b6bbe83075 [sys.catalog]: This master's current role is: LEADER
I20260929 19:04:37.995388 20126 sys_catalog.cc:455] T 00000000000000000000000000000000 P 164ebd7f083c4d4dbc5d04b6bbe83075 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "164ebd7f083c4d4dbc5d04b6bbe83075" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "164ebd7f083c4d4dbc5d04b6bbe83075" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44891 } } }
I20260929 19:04:37.995460 20126 sys_catalog.cc:458] T 00000000000000000000000000000000 P 164ebd7f083c4d4dbc5d04b6bbe83075 [sys.catalog]: This master's current role is: LEADER
W20260929 19:04:37.996436 20141 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 164ebd7f083c4d4dbc5d04b6bbe83075: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 19:04:37.996493 20141 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 19:04:37.996563 20132 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 19:04:37.996687 20132 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 19:04:37.998169 20124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:37.992895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43548 (local address 127.0.98.62:44891)
0929 19:04:37.993097 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0929 19:04:37.993105 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 19:04:37.993134 (+ 29us) server_negotiation.cc:407] Connection header received
0929 19:04:37.993227 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:04:37.993233 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:04:37.993294 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:04:37.993475 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:04:37.993667 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:37.995083 (+ 1416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:37.996236 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:37.996746 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:37.996892 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:04:37.997500 (+ 608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:04:37.997504 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:04:37.997504 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 19:04:37.997686 (+ 182us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:04:37.997725 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 19:04:37.997738 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:04:37.997808 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 19:04:37.997894 (+ 86us) server_negotiation.cc:299] Negotiation successful
0929 19:04:37.997966 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":87,"threads_started":1}
I20260929 19:04:37.998205 20132 catalog_manager.cc:1347] Generated new cluster ID: 2237ef1be1f0446e8245ef15e65f93e7
I20260929 19:04:37.998270 20132 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 19:04:38.017493 20132 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 19:04:38.018124 20132 catalog_manager.cc:1504] Loading token signing keys...
I20260929 19:04:38.027112 20132 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 164ebd7f083c4d4dbc5d04b6bbe83075: Generated new TSK 0
I20260929 19:04:38.027274 20132 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 19:04:38.032297 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUbnLHE/build/release/bin/kudu
/tmp/dist-test-taskUbnLHE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708447806542-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708447806542-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708447806542-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708447806542-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44891
--builtin_ntp_servers=127.0.98.20:46157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 19:04:38.105993 20144 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:04:38.106140 20144 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:04:38.106158 20144 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:04:38.107259 20144 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:04:38.107295 20144 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:04:38.107319 20144 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260929 19:04:38.108634 20144 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708447806542-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708447806542-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708447806542-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44891
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20144
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708447806542-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 18:09:13 UTC on ccycloud.quasar-hpkalf.root.comops.site
build id 4680913
I20260929 19:04:38.108841 20144 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:04:38.109110 20144 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:04:38.113453 20150 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:04:38.113633 20144 server_base.cc:1034] running on GCE node
W20260929 19:04:38.113469 20149 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:04:38.113569 20152 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:04:38.113940 20144 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:04:38.114167 20144 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:04:38.115316 20144 hybrid_clock.cc:648] HybridClock initialized: now 1790708678115301 us; error 26 us; skew 500 ppm
I20260929 19:04:38.116330 20144 webserver.cc:460] Webserver started at http://127.0.98.1:40827/ using document root <none> and password file <none>
I20260929 19:04:38.116545 20144 fs_manager.cc:359] Metadata directory not provided
I20260929 19:04:38.116626 20144 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:04:38.116767 20144 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:04:38.117736 20144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708447806542-392-0/minicluster-data/ts-0/data/instance:
uuid: "97890b6ce5424f6d88e4c0ce32b40960"
format_stamp: "Formatted at 2026-09-29 19:04:38 on dist-test-slave-lqkv"
server_key: "e77285565679f18844ef68c3d29deda1"
server_key_iv: "92e87493bce2c203cc58d25934f1cd1b"
server_key_version: "encryptionkey@0"
I20260929 19:04:38.118129 20144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708447806542-392-0/minicluster-data/ts-0/wal/instance:
uuid: "97890b6ce5424f6d88e4c0ce32b40960"
format_stamp: "Formatted at 2026-09-29 19:04:38 on dist-test-slave-lqkv"
server_key: "e77285565679f18844ef68c3d29deda1"
server_key_iv: "92e87493bce2c203cc58d25934f1cd1b"
server_key_version: "encryptionkey@0"
I20260929 19:04:38.119320 20144 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 19:04:38.119956 20158 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:04:38.120106 20144 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 19:04:38.120194 20144 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708447806542-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708447806542-392-0/minicluster-data/ts-0/wal
uuid: "97890b6ce5424f6d88e4c0ce32b40960"
format_stamp: "Formatted at 2026-09-29 19:04:38 on dist-test-slave-lqkv"
server_key: "e77285565679f18844ef68c3d29deda1"
server_key_iv: "92e87493bce2c203cc58d25934f1cd1b"
server_key_version: "encryptionkey@0"
I20260929 19:04:38.120267 20144 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708447806542-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708447806542-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708447806542-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:04:38.155673 20144 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:04:38.155905 20144 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:04:38.156046 20144 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:04:38.156247 20144 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:04:38.156530 20144 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:04:38.156599 20144 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:04:38.156637 20144 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:04:38.156665 20144 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:04:38.162074 20144 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45815
I20260929 19:04:38.162115 20271 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45815 every 8 connection(s)
I20260929 19:04:38.162391 20144 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708447806542-392-0/minicluster-data/ts-0/data/info.pb
I20260929 19:04:38.166785 20124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:38.162988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50447 (local address 127.0.98.62:44891)
0929 19:04:38.163058 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0929 19:04:38.163060 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:04:38.163252 (+ 192us) server_negotiation.cc:407] Connection header received
0929 19:04:38.163659 (+ 407us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:04:38.163660 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:04:38.163671 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:04:38.163718 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:04:38.164265 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:38.164745 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:38.165674 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:38.165850 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:38.165910 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:04:38.166569 (+ 659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:04:38.166570 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:04:38.166570 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 19:04:38.166576 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:04:38.166589 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 19:04:38.166592 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:04:38.166634 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0929 19:04:38.166688 (+ 54us) server_negotiation.cc:299] Negotiation successful
0929 19:04:38.166711 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 19:04:38.166931 20274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:38.162965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44891 (local address 127.0.98.1:50447)
0929 19:04:38.163149 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0929 19:04:38.163178 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0929 19:04:38.163588 (+ 410us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:04:38.163763 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:04:38.163766 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:04:38.163773 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:04:38.164206 (+ 433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:04:38.164210 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:04:38.164866 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:04:38.164866 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:04:38.165605 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:04:38.165607 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:04:38.165931 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:04:38.165931 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:04:38.166027 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:04:38.166438 (+ 411us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 19:04:38.166438 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:04:38.166462 (+ 24us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 19:04:38.166480 (+ 18us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:04:38.166632 (+ 152us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:04:38.166634 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:04:38.166636 (+ 2us) client_negotiation.cc:769] Sending connection context
0929 19:04:38.166679 (+ 43us) client_negotiation.cc:240] Negotiation successful
0929 19:04:38.166743 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":111,"threads_started":1}
I20260929 19:04:38.166957 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUbnLHE/build/release/bin/kudu as pid 20144
I20260929 19:04:38.167084 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708447806542-392-0/minicluster-data/ts-0/wal/instance
I20260929 19:04:38.167259 392 external_mini_cluster.cc:1428] Setting key cd58af7c7c53dba26ec542e9f8b7c78b
I20260929 19:04:38.167377 20272 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44891
I20260929 19:04:38.167481 20272 heartbeater.cc:464] Registering TS with master...
I20260929 19:04:38.167658 20272 heartbeater.cc:511] Master 127.0.98.62:44891 requested a full tablet report, sending...
I20260929 19:04:38.168186 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUbnLHE/build/release/bin/kudu
/tmp/dist-test-taskUbnLHE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708447806542-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708447806542-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708447806542-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708447806542-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44891
--builtin_ntp_servers=127.0.98.20:46157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 19:04:38.168213 20084 ts_manager.cc:194] Registered new tserver with Master: 97890b6ce5424f6d88e4c0ce32b40960 (127.0.98.1:45815)
I20260929 19:04:38.168936 20084 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:50447
W20260929 19:04:38.241158 20275 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:04:38.241322 20275 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:04:38.241341 20275 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:04:38.242458 20275 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:04:38.242494 20275 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:04:38.242516 20275 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260929 19:04:38.243705 20275 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708447806542-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708447806542-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708447806542-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44891
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20275
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708447806542-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 18:09:13 UTC on ccycloud.quasar-hpkalf.root.comops.site
build id 4680913
I20260929 19:04:38.243944 20275 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:04:38.244208 20275 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:04:38.248776 20283 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:04:38.248831 20275 server_base.cc:1034] running on GCE node
W20260929 19:04:38.248894 20280 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:04:38.248908 20281 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:04:38.249171 20275 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:04:38.249395 20275 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:04:38.250561 20275 hybrid_clock.cc:648] HybridClock initialized: now 1790708678250552 us; error 24 us; skew 500 ppm
I20260929 19:04:38.251696 20275 webserver.cc:460] Webserver started at http://127.0.98.2:42189/ using document root <none> and password file <none>
I20260929 19:04:38.251924 20275 fs_manager.cc:359] Metadata directory not provided
I20260929 19:04:38.251991 20275 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:04:38.252106 20275 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:04:38.253046 20275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708447806542-392-0/minicluster-data/ts-1/data/instance:
uuid: "374ee6d3581f4e70829958bba908ead5"
format_stamp: "Formatted at 2026-09-29 19:04:38 on dist-test-slave-lqkv"
server_key: "56608a07282688fdc8f8fa468a11c480"
server_key_iv: "d9caed24fd6cbdef032eeb6e5b2a76eb"
server_key_version: "encryptionkey@0"
I20260929 19:04:38.253345 20275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708447806542-392-0/minicluster-data/ts-1/wal/instance:
uuid: "374ee6d3581f4e70829958bba908ead5"
format_stamp: "Formatted at 2026-09-29 19:04:38 on dist-test-slave-lqkv"
server_key: "56608a07282688fdc8f8fa468a11c480"
server_key_iv: "d9caed24fd6cbdef032eeb6e5b2a76eb"
server_key_version: "encryptionkey@0"
I20260929 19:04:38.254434 20275 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 19:04:38.255044 20289 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:04:38.255187 20275 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 19:04:38.255271 20275 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708447806542-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708447806542-392-0/minicluster-data/ts-1/wal
uuid: "374ee6d3581f4e70829958bba908ead5"
format_stamp: "Formatted at 2026-09-29 19:04:38 on dist-test-slave-lqkv"
server_key: "56608a07282688fdc8f8fa468a11c480"
server_key_iv: "d9caed24fd6cbdef032eeb6e5b2a76eb"
server_key_version: "encryptionkey@0"
I20260929 19:04:38.255342 20275 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708447806542-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708447806542-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708447806542-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:04:38.266922 20275 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:04:38.267163 20275 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:04:38.267325 20275 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:04:38.267536 20275 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:04:38.267820 20275 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:04:38.267921 20275 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:04:38.267976 20275 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:04:38.268005 20275 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:04:38.273289 20275 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45873
I20260929 19:04:38.273331 20402 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45873 every 8 connection(s)
I20260929 19:04:38.273583 20275 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708447806542-392-0/minicluster-data/ts-1/data/info.pb
I20260929 19:04:38.278165 20124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:38.274163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50457 (local address 127.0.98.62:44891)
0929 19:04:38.274207 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0929 19:04:38.274209 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:04:38.274429 (+ 220us) server_negotiation.cc:407] Connection header received
0929 19:04:38.274891 (+ 462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:04:38.274892 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:04:38.274904 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:04:38.274957 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:04:38.275546 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:38.275998 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:38.276936 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:38.277130 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:38.277203 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:04:38.277895 (+ 692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:04:38.277897 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:04:38.277897 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 19:04:38.277901 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:04:38.277913 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 19:04:38.277917 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:04:38.277975 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0929 19:04:38.278071 (+ 96us) server_negotiation.cc:299] Negotiation successful
0929 19:04:38.278096 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260929 19:04:38.278326 20405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:38.274149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44891 (local address 127.0.98.2:50457)
0929 19:04:38.274317 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0929 19:04:38.274342 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 19:04:38.274780 (+ 438us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:04:38.275013 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:04:38.275017 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:04:38.275025 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:04:38.275462 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:04:38.275465 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:04:38.276081 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:04:38.276082 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:04:38.276808 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:04:38.276810 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:04:38.277209 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:04:38.277210 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:04:38.277335 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:04:38.277742 (+ 407us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 19:04:38.277743 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:04:38.277765 (+ 22us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 19:04:38.277784 (+ 19us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:04:38.277975 (+ 191us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:04:38.277981 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:04:38.277984 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 19:04:38.278060 (+ 76us) client_negotiation.cc:240] Negotiation successful
0929 19:04:38.278145 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":71,"threads_started":1}
I20260929 19:04:38.278714 20403 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44891
I20260929 19:04:38.278775 20403 heartbeater.cc:464] Registering TS with master...
I20260929 19:04:38.278919 20403 heartbeater.cc:511] Master 127.0.98.62:44891 requested a full tablet report, sending...
I20260929 19:04:38.279352 20084 ts_manager.cc:194] Registered new tserver with Master: 374ee6d3581f4e70829958bba908ead5 (127.0.98.2:45873)
I20260929 19:04:38.279673 20084 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:50457
I20260929 19:04:38.282400 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUbnLHE/build/release/bin/kudu as pid 20275
I20260929 19:04:38.282514 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708447806542-392-0/minicluster-data/ts-1/wal/instance
I20260929 19:04:38.282688 392 external_mini_cluster.cc:1428] Setting key 7c4aa02d020ca2d7e2d2d06ca03beeaa
I20260929 19:04:38.283439 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUbnLHE/build/release/bin/kudu
/tmp/dist-test-taskUbnLHE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708447806542-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708447806542-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708447806542-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708447806542-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44891
--builtin_ntp_servers=127.0.98.20:46157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 19:04:38.356050 20406 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:04:38.356208 20406 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:04:38.356227 20406 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:04:38.357507 20406 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:04:38.357566 20406 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:04:38.357609 20406 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260929 19:04:38.358834 20406 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708447806542-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708447806542-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708447806542-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44891
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20406
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708447806542-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 18:09:13 UTC on ccycloud.quasar-hpkalf.root.comops.site
build id 4680913
I20260929 19:04:38.359050 20406 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:04:38.359299 20406 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:04:38.363552 20411 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:04:38.363574 20414 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:04:38.363562 20412 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:04:38.363714 20406 server_base.cc:1034] running on GCE node
I20260929 19:04:38.363960 20406 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:04:38.364193 20406 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:04:38.365330 20406 hybrid_clock.cc:648] HybridClock initialized: now 1790708678365314 us; error 27 us; skew 500 ppm
I20260929 19:04:38.366338 20406 webserver.cc:460] Webserver started at http://127.0.98.3:39637/ using document root <none> and password file <none>
I20260929 19:04:38.366544 20406 fs_manager.cc:359] Metadata directory not provided
I20260929 19:04:38.366626 20406 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:04:38.366752 20406 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:04:38.367722 20406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708447806542-392-0/minicluster-data/ts-2/data/instance:
uuid: "0e9822a8cbeb411eaec780a6b58a0ac3"
format_stamp: "Formatted at 2026-09-29 19:04:38 on dist-test-slave-lqkv"
server_key: "1caad38cddc936bed91c34d5f1aa9152"
server_key_iv: "829bc27c76e0cc066fe4be861870093e"
server_key_version: "encryptionkey@0"
I20260929 19:04:38.368098 20406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708447806542-392-0/minicluster-data/ts-2/wal/instance:
uuid: "0e9822a8cbeb411eaec780a6b58a0ac3"
format_stamp: "Formatted at 2026-09-29 19:04:38 on dist-test-slave-lqkv"
server_key: "1caad38cddc936bed91c34d5f1aa9152"
server_key_iv: "829bc27c76e0cc066fe4be861870093e"
server_key_version: "encryptionkey@0"
I20260929 19:04:38.369316 20406 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 19:04:38.369915 20420 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:04:38.370044 20406 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 19:04:38.370126 20406 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708447806542-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708447806542-392-0/minicluster-data/ts-2/wal
uuid: "0e9822a8cbeb411eaec780a6b58a0ac3"
format_stamp: "Formatted at 2026-09-29 19:04:38 on dist-test-slave-lqkv"
server_key: "1caad38cddc936bed91c34d5f1aa9152"
server_key_iv: "829bc27c76e0cc066fe4be861870093e"
server_key_version: "encryptionkey@0"
I20260929 19:04:38.370195 20406 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708447806542-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708447806542-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708447806542-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:04:38.380380 20406 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:04:38.380625 20406 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:04:38.380753 20406 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:04:38.380966 20406 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 19:04:38.381254 20406 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 19:04:38.381312 20406 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 19:04:38.381348 20406 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 19:04:38.381374 20406 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 19:04:38.386364 20406 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41895
I20260929 19:04:38.386427 20533 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41895 every 8 connection(s)
I20260929 19:04:38.386711 20406 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708447806542-392-0/minicluster-data/ts-2/data/info.pb
I20260929 19:04:38.387468 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUbnLHE/build/release/bin/kudu as pid 20406
I20260929 19:04:38.387591 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708447806542-392-0/minicluster-data/ts-2/wal/instance
I20260929 19:04:38.387761 392 external_mini_cluster.cc:1428] Setting key 3680f9a6f7e31c94f3361effdb80bb78
I20260929 19:04:38.391577 20124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:38.387340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38061 (local address 127.0.98.62:44891)
0929 19:04:38.387391 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0929 19:04:38.387393 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:04:38.387620 (+ 227us) server_negotiation.cc:407] Connection header received
0929 19:04:38.388180 (+ 560us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:04:38.388181 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:04:38.388191 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:04:38.388244 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:04:38.389010 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:38.389467 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:38.390327 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:38.390525 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:38.390600 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:04:38.391339 (+ 739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:04:38.391340 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:04:38.391341 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 19:04:38.391345 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:04:38.391365 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 19:04:38.391369 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:04:38.391420 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0929 19:04:38.391482 (+ 62us) server_negotiation.cc:299] Negotiation successful
0929 19:04:38.391509 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260929 19:04:38.391757 20536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:38.387321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44891 (local address 127.0.98.3:38061)
0929 19:04:38.387498 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0929 19:04:38.387547 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 19:04:38.388096 (+ 549us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:04:38.388308 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:04:38.388312 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:04:38.388322 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:04:38.388931 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:04:38.388936 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:04:38.389558 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:04:38.389559 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:04:38.390242 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:04:38.390244 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:04:38.390615 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:04:38.390616 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:04:38.390739 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:04:38.391198 (+ 459us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 19:04:38.391199 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:04:38.391221 (+ 22us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 19:04:38.391238 (+ 17us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:04:38.391421 (+ 183us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:04:38.391426 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:04:38.391428 (+ 2us) client_negotiation.cc:769] Sending connection context
0929 19:04:38.391485 (+ 57us) client_negotiation.cc:240] Negotiation successful
0929 19:04:38.391566 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":91,"threads_started":1}
I20260929 19:04:38.392140 20534 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44891
I20260929 19:04:38.392215 20534 heartbeater.cc:464] Registering TS with master...
I20260929 19:04:38.392328 20534 heartbeater.cc:511] Master 127.0.98.62:44891 requested a full tablet report, sending...
I20260929 19:04:38.392609 20084 ts_manager.cc:194] Registered new tserver with Master: 0e9822a8cbeb411eaec780a6b58a0ac3 (127.0.98.3:41895)
I20260929 19:04:38.392984 20084 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:38061
I20260929 19:04:38.399602 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 19:04:38.402987 20124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:38.400773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43556 (local address 127.0.98.62:44891)
0929 19:04:38.400829 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0929 19:04:38.400831 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:04:38.400955 (+ 124us) server_negotiation.cc:407] Connection header received
0929 19:04:38.400988 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:04:38.400988 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:04:38.400997 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:04:38.401030 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:04:38.401158 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:38.401549 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:38.402178 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:38.402325 (+ 147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:38.402371 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:04:38.402805 (+ 434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:04:38.402806 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:04:38.402806 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 19:04:38.402809 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:04:38.402819 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 19:04:38.402822 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:04:38.402861 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0929 19:04:38.402892 (+ 31us) server_negotiation.cc:299] Negotiation successful
0929 19:04:38.402908 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260929 19:04:38.404145 20084 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43556:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 19:04:38.405577 20084 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 19:04:38.411077 20543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:38.407454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41895 (local address 127.0.0.1:56534)
0929 19:04:38.407756 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0929 19:04:38.407802 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 19:04:38.407882 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:04:38.408103 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:04:38.408106 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:04:38.408108 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:04:38.408166 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:04:38.408167 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:04:38.409110 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:04:38.409111 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:04:38.410043 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:04:38.410045 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:04:38.410841 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:04:38.410842 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:04:38.410898 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:04:38.410905 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 19:04:38.410971 (+ 66us) client_negotiation.cc:240] Negotiation successful
0929 19:04:38.411014 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"mutex_wait_us":3,"thread_start_us":73,"threads_started":1}
I20260929 19:04:38.411090 20544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:38.407526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56534 (local address 127.0.98.3:41895)
0929 19:04:38.407662 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0929 19:04:38.407667 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 19:04:38.407858 (+ 191us) server_negotiation.cc:407] Connection header received
0929 19:04:38.407992 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:04:38.407994 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:04:38.408008 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:04:38.408084 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:04:38.408254 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:38.409042 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:38.410131 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:38.410721 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:38.410750 (+ 29us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:04:38.410798 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0929 19:04:38.411002 (+ 204us) server_negotiation.cc:299] Negotiation successful
0929 19:04:38.411031 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":74,"threads_started":1}
I20260929 19:04:38.411773 20545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:38.407620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57010 (local address 127.0.98.1:45815)
0929 19:04:38.407775 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0929 19:04:38.407780 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 19:04:38.407984 (+ 204us) server_negotiation.cc:407] Connection header received
0929 19:04:38.408030 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:04:38.408033 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:04:38.408050 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:04:38.408194 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:04:38.408394 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:38.409477 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:38.410597 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:38.411487 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:38.411540 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:04:38.411585 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0929 19:04:38.411695 (+ 110us) server_negotiation.cc:299] Negotiation successful
0929 19:04:38.411721 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":88,"threads_started":1}
I20260929 19:04:38.411787 20546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:38.407454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45815 (local address 127.0.0.1:57010)
0929 19:04:38.407754 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0929 19:04:38.407801 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 19:04:38.407866 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:04:38.408260 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:04:38.408261 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:04:38.408264 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:04:38.408307 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:04:38.408309 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:04:38.409578 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:04:38.409579 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:04:38.410526 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:04:38.410528 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:04:38.411564 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:04:38.411564 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:04:38.411614 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:04:38.411617 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 19:04:38.411687 (+ 70us) client_negotiation.cc:240] Negotiation successful
0929 19:04:38.411722 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"mutex_wait_us":3,"thread_start_us":62,"threads_started":1}
I20260929 19:04:38.411772 20468 tablet_service.cc:1467] Processing CreateTablet for tablet af2f85c6a6e749cfb7b181d0aeab9c0d (DEFAULT_TABLE table=test-table [id=c0d5be69a733488fa2bcba59a037b426]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:04:38.412102 20468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af2f85c6a6e749cfb7b181d0aeab9c0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:04:38.412420 20206 tablet_service.cc:1467] Processing CreateTablet for tablet af2f85c6a6e749cfb7b181d0aeab9c0d (DEFAULT_TABLE table=test-table [id=c0d5be69a733488fa2bcba59a037b426]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:04:38.412707 20206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af2f85c6a6e749cfb7b181d0aeab9c0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:04:38.415215 20548 tablet_bootstrap.cc:492] T af2f85c6a6e749cfb7b181d0aeab9c0d P 0e9822a8cbeb411eaec780a6b58a0ac3: Bootstrap starting.
I20260929 19:04:38.415328 20547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:38.407868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43154 (local address 127.0.98.2:45873)
0929 19:04:38.408188 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0929 19:04:38.408192 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 19:04:38.408201 (+ 9us) server_negotiation.cc:407] Connection header received
0929 19:04:38.408237 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:04:38.408239 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:04:38.408254 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:04:38.408324 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:04:38.408541 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:38.409548 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:38.410465 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:38.411123 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:38.411303 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:04:38.411394 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0929 19:04:38.415208 (+ 3814us) server_negotiation.cc:299] Negotiation successful
0929 19:04:38.415249 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":116,"threads_started":1}
I20260929 19:04:38.415553 20542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:38.407645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45873 (local address 127.0.0.1:43154)
0929 19:04:38.407995 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0929 19:04:38.408004 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0929 19:04:38.408038 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:04:38.408412 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:04:38.408413 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:04:38.408415 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:04:38.408459 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:04:38.408461 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:04:38.409612 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:04:38.409613 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:04:38.410390 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:04:38.410392 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:04:38.411315 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:04:38.411317 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:04:38.411373 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:04:38.411376 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 19:04:38.415454 (+ 4078us) client_negotiation.cc:240] Negotiation successful
0929 19:04:38.415496 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":63,"threads_started":1}
I20260929 19:04:38.415899 20548 tablet_bootstrap.cc:654] T af2f85c6a6e749cfb7b181d0aeab9c0d P 0e9822a8cbeb411eaec780a6b58a0ac3: Neither blocks nor log segments found. Creating new log.
I20260929 19:04:38.415951 20549 tablet_bootstrap.cc:492] T af2f85c6a6e749cfb7b181d0aeab9c0d P 97890b6ce5424f6d88e4c0ce32b40960: Bootstrap starting.
I20260929 19:04:38.416167 20548 log.cc:824] T af2f85c6a6e749cfb7b181d0aeab9c0d P 0e9822a8cbeb411eaec780a6b58a0ac3: Log is configured to *not* fsync() on all Append() calls
I20260929 19:04:38.416180 20337 tablet_service.cc:1467] Processing CreateTablet for tablet af2f85c6a6e749cfb7b181d0aeab9c0d (DEFAULT_TABLE table=test-table [id=c0d5be69a733488fa2bcba59a037b426]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 19:04:38.416433 20549 tablet_bootstrap.cc:654] T af2f85c6a6e749cfb7b181d0aeab9c0d P 97890b6ce5424f6d88e4c0ce32b40960: Neither blocks nor log segments found. Creating new log.
I20260929 19:04:38.416455 20337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af2f85c6a6e749cfb7b181d0aeab9c0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:04:38.416721 20549 log.cc:824] T af2f85c6a6e749cfb7b181d0aeab9c0d P 97890b6ce5424f6d88e4c0ce32b40960: Log is configured to *not* fsync() on all Append() calls
I20260929 19:04:38.416923 20548 tablet_bootstrap.cc:492] T af2f85c6a6e749cfb7b181d0aeab9c0d P 0e9822a8cbeb411eaec780a6b58a0ac3: No bootstrap required, opened a new log
I20260929 19:04:38.416980 20548 ts_tablet_manager.cc:1397] T af2f85c6a6e749cfb7b181d0aeab9c0d P 0e9822a8cbeb411eaec780a6b58a0ac3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260929 19:04:38.417346 20549 tablet_bootstrap.cc:492] T af2f85c6a6e749cfb7b181d0aeab9c0d P 97890b6ce5424f6d88e4c0ce32b40960: No bootstrap required, opened a new log
I20260929 19:04:38.417387 20549 ts_tablet_manager.cc:1397] T af2f85c6a6e749cfb7b181d0aeab9c0d P 97890b6ce5424f6d88e4c0ce32b40960: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260929 19:04:38.419008 20549 raft_consensus.cc:348] T af2f85c6a6e749cfb7b181d0aeab9c0d P 97890b6ce5424f6d88e4c0ce32b40960 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e9822a8cbeb411eaec780a6b58a0ac3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41895 } } peers { permanent_uuid: "97890b6ce5424f6d88e4c0ce32b40960" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45815 } } peers { permanent_uuid: "374ee6d3581f4e70829958bba908ead5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45873 } }
I20260929 19:04:38.419108 20549 raft_consensus.cc:374] T af2f85c6a6e749cfb7b181d0aeab9c0d P 97890b6ce5424f6d88e4c0ce32b40960 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:04:38.419126 20549 raft_consensus.cc:729] T af2f85c6a6e749cfb7b181d0aeab9c0d P 97890b6ce5424f6d88e4c0ce32b40960 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97890b6ce5424f6d88e4c0ce32b40960, State: Initialized, Role: FOLLOWER
I20260929 19:04:38.419227 20549 consensus_queue.cc:260] T af2f85c6a6e749cfb7b181d0aeab9c0d P 97890b6ce5424f6d88e4c0ce32b40960 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e9822a8cbeb411eaec780a6b58a0ac3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41895 } } peers { permanent_uuid: "97890b6ce5424f6d88e4c0ce32b40960" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45815 } } peers { permanent_uuid: "374ee6d3581f4e70829958bba908ead5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45873 } }
I20260929 19:04:38.419348 20548 raft_consensus.cc:348] T af2f85c6a6e749cfb7b181d0aeab9c0d P 0e9822a8cbeb411eaec780a6b58a0ac3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e9822a8cbeb411eaec780a6b58a0ac3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41895 } } peers { permanent_uuid: "97890b6ce5424f6d88e4c0ce32b40960" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45815 } } peers { permanent_uuid: "374ee6d3581f4e70829958bba908ead5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45873 } }
I20260929 19:04:38.419449 20549 ts_tablet_manager.cc:1428] T af2f85c6a6e749cfb7b181d0aeab9c0d P 97890b6ce5424f6d88e4c0ce32b40960: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260929 19:04:38.419478 20272 heartbeater.cc:503] Master 127.0.98.62:44891 was elected leader, sending a full tablet report...
I20260929 19:04:38.419490 20548 raft_consensus.cc:374] T af2f85c6a6e749cfb7b181d0aeab9c0d P 0e9822a8cbeb411eaec780a6b58a0ac3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:04:38.419520 20548 raft_consensus.cc:729] T af2f85c6a6e749cfb7b181d0aeab9c0d P 0e9822a8cbeb411eaec780a6b58a0ac3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e9822a8cbeb411eaec780a6b58a0ac3, State: Initialized, Role: FOLLOWER
I20260929 19:04:38.419612 20548 consensus_queue.cc:260] T af2f85c6a6e749cfb7b181d0aeab9c0d P 0e9822a8cbeb411eaec780a6b58a0ac3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e9822a8cbeb411eaec780a6b58a0ac3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41895 } } peers { permanent_uuid: "97890b6ce5424f6d88e4c0ce32b40960" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45815 } } peers { permanent_uuid: "374ee6d3581f4e70829958bba908ead5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45873 } }
I20260929 19:04:38.419839 20548 ts_tablet_manager.cc:1428] T af2f85c6a6e749cfb7b181d0aeab9c0d P 0e9822a8cbeb411eaec780a6b58a0ac3: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 19:04:38.419860 20534 heartbeater.cc:503] Master 127.0.98.62:44891 was elected leader, sending a full tablet report...
I20260929 19:04:38.420650 20554 tablet_bootstrap.cc:492] T af2f85c6a6e749cfb7b181d0aeab9c0d P 374ee6d3581f4e70829958bba908ead5: Bootstrap starting.
I20260929 19:04:38.421173 20554 tablet_bootstrap.cc:654] T af2f85c6a6e749cfb7b181d0aeab9c0d P 374ee6d3581f4e70829958bba908ead5: Neither blocks nor log segments found. Creating new log.
I20260929 19:04:38.421375 20554 log.cc:824] T af2f85c6a6e749cfb7b181d0aeab9c0d P 374ee6d3581f4e70829958bba908ead5: Log is configured to *not* fsync() on all Append() calls
I20260929 19:04:38.421875 20554 tablet_bootstrap.cc:492] T af2f85c6a6e749cfb7b181d0aeab9c0d P 374ee6d3581f4e70829958bba908ead5: No bootstrap required, opened a new log
I20260929 19:04:38.421921 20554 ts_tablet_manager.cc:1397] T af2f85c6a6e749cfb7b181d0aeab9c0d P 374ee6d3581f4e70829958bba908ead5: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260929 19:04:38.423476 20554 raft_consensus.cc:348] T af2f85c6a6e749cfb7b181d0aeab9c0d P 374ee6d3581f4e70829958bba908ead5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e9822a8cbeb411eaec780a6b58a0ac3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41895 } } peers { permanent_uuid: "97890b6ce5424f6d88e4c0ce32b40960" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45815 } } peers { permanent_uuid: "374ee6d3581f4e70829958bba908ead5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45873 } }
I20260929 19:04:38.423588 20554 raft_consensus.cc:374] T af2f85c6a6e749cfb7b181d0aeab9c0d P 374ee6d3581f4e70829958bba908ead5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:04:38.423607 20554 raft_consensus.cc:729] T af2f85c6a6e749cfb7b181d0aeab9c0d P 374ee6d3581f4e70829958bba908ead5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 374ee6d3581f4e70829958bba908ead5, State: Initialized, Role: FOLLOWER
I20260929 19:04:38.423691 20554 consensus_queue.cc:260] T af2f85c6a6e749cfb7b181d0aeab9c0d P 374ee6d3581f4e70829958bba908ead5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e9822a8cbeb411eaec780a6b58a0ac3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41895 } } peers { permanent_uuid: "97890b6ce5424f6d88e4c0ce32b40960" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45815 } } peers { permanent_uuid: "374ee6d3581f4e70829958bba908ead5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45873 } }
I20260929 19:04:38.423916 20554 ts_tablet_manager.cc:1428] T af2f85c6a6e749cfb7b181d0aeab9c0d P 374ee6d3581f4e70829958bba908ead5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 19:04:38.423964 20403 heartbeater.cc:503] Master 127.0.98.62:44891 was elected leader, sending a full tablet report...
I20260929 19:04:38.468649 20552 raft_consensus.cc:482] T af2f85c6a6e749cfb7b181d0aeab9c0d P 97890b6ce5424f6d88e4c0ce32b40960 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 19:04:38.468745 20552 raft_consensus.cc:504] T af2f85c6a6e749cfb7b181d0aeab9c0d P 97890b6ce5424f6d88e4c0ce32b40960 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e9822a8cbeb411eaec780a6b58a0ac3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41895 } } peers { permanent_uuid: "97890b6ce5424f6d88e4c0ce32b40960" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45815 } } peers { permanent_uuid: "374ee6d3581f4e70829958bba908ead5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45873 } }
I20260929 19:04:38.469089 20552 leader_election.cc:288] T af2f85c6a6e749cfb7b181d0aeab9c0d P 97890b6ce5424f6d88e4c0ce32b40960 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0e9822a8cbeb411eaec780a6b58a0ac3 (127.0.98.3:41895), 374ee6d3581f4e70829958bba908ead5 (127.0.98.2:45873)
I20260929 19:04:38.471498 20274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:38.469233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41895 (local address 127.0.98.1:44527)
0929 19:04:38.469294 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0929 19:04:38.469306 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 19:04:38.469368 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:04:38.469490 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:04:38.469491 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:04:38.469492 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:04:38.469558 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:04:38.469560 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:04:38.470121 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:04:38.470122 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:04:38.470773 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:04:38.470775 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:04:38.471247 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:04:38.471247 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:04:38.471312 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:04:38.471315 (+ 3us) client_negotiation.cc:769] Sending connection context
0929 19:04:38.471394 (+ 79us) client_negotiation.cc:240] Negotiation successful
0929 19:04:38.471433 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260929 19:04:38.471498 20544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:38.469327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44527 (local address 127.0.98.3:41895)
0929 19:04:38.469384 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0929 19:04:38.469387 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 19:04:38.469396 (+ 9us) server_negotiation.cc:407] Connection header received
0929 19:04:38.469424 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:04:38.469425 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:04:38.469434 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:04:38.469485 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:04:38.469621 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:38.470045 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:38.470863 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:38.471175 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:38.471247 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:04:38.471256 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0929 19:04:38.471399 (+ 143us) server_negotiation.cc:299] Negotiation successful
0929 19:04:38.471433 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260929 19:04:38.471752 20557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:38.469341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45873 (local address 127.0.98.1:36563)
0929 19:04:38.469506 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0929 19:04:38.469514 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0929 19:04:38.469581 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:04:38.469685 (+ 104us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:04:38.469686 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:04:38.469687 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 19:04:38.469750 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:04:38.469751 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:04:38.470303 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:04:38.470304 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:04:38.470957 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:04:38.470959 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:04:38.471479 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:04:38.471480 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:04:38.471545 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:04:38.471549 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 19:04:38.471634 (+ 85us) client_negotiation.cc:240] Negotiation successful
0929 19:04:38.471674 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":102,"threads_started":1}
I20260929 19:04:38.471756 20547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:38.469427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36563 (local address 127.0.98.2:45873)
0929 19:04:38.469481 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0929 19:04:38.469482 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 19:04:38.469570 (+ 88us) server_negotiation.cc:407] Connection header received
0929 19:04:38.469630 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:04:38.469631 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:04:38.469639 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:04:38.469684 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 19:04:38.469815 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:38.470228 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:38.471039 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:38.471391 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:38.471470 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:04:38.471479 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0929 19:04:38.471648 (+ 169us) server_negotiation.cc:299] Negotiation successful
0929 19:04:38.471696 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260929 19:04:38.471781 20488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af2f85c6a6e749cfb7b181d0aeab9c0d" candidate_uuid: "97890b6ce5424f6d88e4c0ce32b40960" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e9822a8cbeb411eaec780a6b58a0ac3" is_pre_election: true
I20260929 19:04:38.471903 20488 raft_consensus.cc:2454] T af2f85c6a6e749cfb7b181d0aeab9c0d P 0e9822a8cbeb411eaec780a6b58a0ac3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97890b6ce5424f6d88e4c0ce32b40960 in term 0.
I20260929 19:04:38.472131 20162 leader_election.cc:302] T af2f85c6a6e749cfb7b181d0aeab9c0d P 97890b6ce5424f6d88e4c0ce32b40960 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e9822a8cbeb411eaec780a6b58a0ac3, 97890b6ce5424f6d88e4c0ce32b40960; no voters:
I20260929 19:04:38.472239 20357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af2f85c6a6e749cfb7b181d0aeab9c0d" candidate_uuid: "97890b6ce5424f6d88e4c0ce32b40960" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "374ee6d3581f4e70829958bba908ead5" is_pre_election: true
I20260929 19:04:38.472395 20357 raft_consensus.cc:2454] T af2f85c6a6e749cfb7b181d0aeab9c0d P 374ee6d3581f4e70829958bba908ead5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97890b6ce5424f6d88e4c0ce32b40960 in term 0.
I20260929 19:04:38.472520 20552 raft_consensus.cc:2781] T af2f85c6a6e749cfb7b181d0aeab9c0d P 97890b6ce5424f6d88e4c0ce32b40960 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 19:04:38.472581 20552 raft_consensus.cc:482] T af2f85c6a6e749cfb7b181d0aeab9c0d P 97890b6ce5424f6d88e4c0ce32b40960 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 19:04:38.472615 20552 raft_consensus.cc:3037] T af2f85c6a6e749cfb7b181d0aeab9c0d P 97890b6ce5424f6d88e4c0ce32b40960 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:04:38.473395 20552 raft_consensus.cc:504] T af2f85c6a6e749cfb7b181d0aeab9c0d P 97890b6ce5424f6d88e4c0ce32b40960 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e9822a8cbeb411eaec780a6b58a0ac3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41895 } } peers { permanent_uuid: "97890b6ce5424f6d88e4c0ce32b40960" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45815 } } peers { permanent_uuid: "374ee6d3581f4e70829958bba908ead5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45873 } }
I20260929 19:04:38.473508 20552 leader_election.cc:288] T af2f85c6a6e749cfb7b181d0aeab9c0d P 97890b6ce5424f6d88e4c0ce32b40960 [CANDIDATE]: Term 1 election: Requested vote from peers 0e9822a8cbeb411eaec780a6b58a0ac3 (127.0.98.3:41895), 374ee6d3581f4e70829958bba908ead5 (127.0.98.2:45873)
I20260929 19:04:38.473654 20357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af2f85c6a6e749cfb7b181d0aeab9c0d" candidate_uuid: "97890b6ce5424f6d88e4c0ce32b40960" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "374ee6d3581f4e70829958bba908ead5"
I20260929 19:04:38.473680 20488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af2f85c6a6e749cfb7b181d0aeab9c0d" candidate_uuid: "97890b6ce5424f6d88e4c0ce32b40960" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e9822a8cbeb411eaec780a6b58a0ac3"
I20260929 19:04:38.473711 20357 raft_consensus.cc:3037] T af2f85c6a6e749cfb7b181d0aeab9c0d P 374ee6d3581f4e70829958bba908ead5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:04:38.473733 20488 raft_consensus.cc:3037] T af2f85c6a6e749cfb7b181d0aeab9c0d P 0e9822a8cbeb411eaec780a6b58a0ac3 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:04:38.474385 20488 raft_consensus.cc:2454] T af2f85c6a6e749cfb7b181d0aeab9c0d P 0e9822a8cbeb411eaec780a6b58a0ac3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97890b6ce5424f6d88e4c0ce32b40960 in term 1.
I20260929 19:04:38.474388 20357 raft_consensus.cc:2454] T af2f85c6a6e749cfb7b181d0aeab9c0d P 374ee6d3581f4e70829958bba908ead5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97890b6ce5424f6d88e4c0ce32b40960 in term 1.
I20260929 19:04:38.474546 20162 leader_election.cc:302] T af2f85c6a6e749cfb7b181d0aeab9c0d P 97890b6ce5424f6d88e4c0ce32b40960 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 374ee6d3581f4e70829958bba908ead5, 97890b6ce5424f6d88e4c0ce32b40960; no voters:
I20260929 19:04:38.474641 20552 raft_consensus.cc:2781] T af2f85c6a6e749cfb7b181d0aeab9c0d P 97890b6ce5424f6d88e4c0ce32b40960 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:04:38.474787 20552 raft_consensus.cc:686] T af2f85c6a6e749cfb7b181d0aeab9c0d P 97890b6ce5424f6d88e4c0ce32b40960 [term 1 LEADER]: Becoming Leader. State: Replica: 97890b6ce5424f6d88e4c0ce32b40960, State: Running, Role: LEADER
I20260929 19:04:38.474886 20552 consensus_queue.cc:237] T af2f85c6a6e749cfb7b181d0aeab9c0d P 97890b6ce5424f6d88e4c0ce32b40960 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e9822a8cbeb411eaec780a6b58a0ac3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41895 } } peers { permanent_uuid: "97890b6ce5424f6d88e4c0ce32b40960" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45815 } } peers { permanent_uuid: "374ee6d3581f4e70829958bba908ead5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45873 } }
I20260929 19:04:38.475586 20083 catalog_manager.cc:5495] T af2f85c6a6e749cfb7b181d0aeab9c0d P 97890b6ce5424f6d88e4c0ce32b40960 reported cstate change: term changed from 0 to 1, leader changed from <none> to 97890b6ce5424f6d88e4c0ce32b40960 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "97890b6ce5424f6d88e4c0ce32b40960" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e9822a8cbeb411eaec780a6b58a0ac3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41895 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97890b6ce5424f6d88e4c0ce32b40960" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45815 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "374ee6d3581f4e70829958bba908ead5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45873 } health_report { overall_health: UNKNOWN } } }
I20260929 19:04:38.484607 20545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:38.482702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57020 (local address 127.0.98.1:45815)
0929 19:04:38.482739 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0929 19:04:38.482741 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:04:38.482748 (+ 7us) server_negotiation.cc:407] Connection header received
0929 19:04:38.482802 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:04:38.482802 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:04:38.482815 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:04:38.482870 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 19:04:38.482960 (+ 90us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:38.483331 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:38.483957 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:38.484158 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:38.484225 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:04:38.484345 (+ 120us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 19:04:38.484421 (+ 76us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 19:04:38.484470 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0929 19:04:38.484503 (+ 33us) server_negotiation.cc:299] Negotiation successful
0929 19:04:38.484517 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260929 19:04:38.486239 20488 raft_consensus.cc:1261] T af2f85c6a6e749cfb7b181d0aeab9c0d P 0e9822a8cbeb411eaec780a6b58a0ac3 [term 1 FOLLOWER]: Refusing update from remote peer 97890b6ce5424f6d88e4c0ce32b40960: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 19:04:38.486244 20357 raft_consensus.cc:1261] T af2f85c6a6e749cfb7b181d0aeab9c0d P 374ee6d3581f4e70829958bba908ead5 [term 1 FOLLOWER]: Refusing update from remote peer 97890b6ce5424f6d88e4c0ce32b40960: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 19:04:38.486544 20552 consensus_queue.cc:1035] T af2f85c6a6e749cfb7b181d0aeab9c0d P 97890b6ce5424f6d88e4c0ce32b40960 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e9822a8cbeb411eaec780a6b58a0ac3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 19:04:38.486680 20558 consensus_queue.cc:1035] T af2f85c6a6e749cfb7b181d0aeab9c0d P 97890b6ce5424f6d88e4c0ce32b40960 [LEADER]: Connected to new peer: Peer: permanent_uuid: "374ee6d3581f4e70829958bba908ead5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 19:04:38.490696 20083 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43556:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 19:04:38.490787 20083 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43556:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 19:04:38.491459 20083 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 164ebd7f083c4d4dbc5d04b6bbe83075: Sending DeleteTablet for 3 replicas of tablet af2f85c6a6e749cfb7b181d0aeab9c0d
I20260929 19:04:38.491720 20468 tablet_service.cc:1514] Processing DeleteTablet for tablet af2f85c6a6e749cfb7b181d0aeab9c0d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:04:38 UTC) from {username='slave'} at 127.0.0.1:56534
I20260929 19:04:38.491765 20337 tablet_service.cc:1514] Processing DeleteTablet for tablet af2f85c6a6e749cfb7b181d0aeab9c0d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:04:38 UTC) from {username='slave'} at 127.0.0.1:43154
I20260929 19:04:38.491748 20206 tablet_service.cc:1514] Processing DeleteTablet for tablet af2f85c6a6e749cfb7b181d0aeab9c0d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 19:04:38 UTC) from {username='slave'} at 127.0.0.1:57010
I20260929 19:04:38.492038 20572 tablet_replica.cc:330] T af2f85c6a6e749cfb7b181d0aeab9c0d P 97890b6ce5424f6d88e4c0ce32b40960: stopping tablet replica
I20260929 19:04:38.492126 20572 raft_consensus.cc:2229] T af2f85c6a6e749cfb7b181d0aeab9c0d P 97890b6ce5424f6d88e4c0ce32b40960 [term 1 LEADER]: Raft consensus shutting down.
I20260929 19:04:38.492182 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUbnLHE/build/release/bin/kudu with pid 20144
I20260929 19:04:38.492209 20572 raft_consensus.cc:2258] T af2f85c6a6e749cfb7b181d0aeab9c0d P 97890b6ce5424f6d88e4c0ce32b40960 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 19:04:38.493573 20573 tablet_replica.cc:330] T af2f85c6a6e749cfb7b181d0aeab9c0d P 374ee6d3581f4e70829958bba908ead5: stopping tablet replica
I20260929 19:04:38.493666 20573 raft_consensus.cc:2229] T af2f85c6a6e749cfb7b181d0aeab9c0d P 374ee6d3581f4e70829958bba908ead5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 19:04:38.493759 20573 raft_consensus.cc:2258] T af2f85c6a6e749cfb7b181d0aeab9c0d P 374ee6d3581f4e70829958bba908ead5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 19:04:38.491956 20571 tablet_replica.cc:330] T af2f85c6a6e749cfb7b181d0aeab9c0d P 0e9822a8cbeb411eaec780a6b58a0ac3: stopping tablet replica
I20260929 19:04:38.493897 20571 raft_consensus.cc:2229] T af2f85c6a6e749cfb7b181d0aeab9c0d P 0e9822a8cbeb411eaec780a6b58a0ac3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 19:04:38.493978 20571 raft_consensus.cc:2258] T af2f85c6a6e749cfb7b181d0aeab9c0d P 0e9822a8cbeb411eaec780a6b58a0ac3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 19:04:38.494318 20571 ts_tablet_manager.cc:1905] T af2f85c6a6e749cfb7b181d0aeab9c0d P 0e9822a8cbeb411eaec780a6b58a0ac3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 19:04:38.496129 20571 ts_tablet_manager.cc:1918] T af2f85c6a6e749cfb7b181d0aeab9c0d P 0e9822a8cbeb411eaec780a6b58a0ac3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 19:04:38.496194 20571 log.cc:1196] T af2f85c6a6e749cfb7b181d0aeab9c0d P 0e9822a8cbeb411eaec780a6b58a0ac3: Deleting WAL directory at /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790708447806542-392-0/minicluster-data/ts-2/wal/wals/af2f85c6a6e749cfb7b181d0aeab9c0d
I20260929 19:04:38.496421 20571 ts_tablet_manager.cc:1939] T af2f85c6a6e749cfb7b181d0aeab9c0d P 0e9822a8cbeb411eaec780a6b58a0ac3: Deleting consensus metadata
I20260929 19:04:38.496892 20071 catalog_manager.cc:4841] TS 0e9822a8cbeb411eaec780a6b58a0ac3 (127.0.98.3:41895): tablet af2f85c6a6e749cfb7b181d0aeab9c0d (table test-table [id=c0d5be69a733488fa2bcba59a037b426]) successfully deleted
W20260929 19:04:38.498318 20070 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:45815 (error 108)
I20260929 19:04:38.498579 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUbnLHE/build/release/bin/kudu with pid 20275
I20260929 19:04:38.498608 20573 ts_tablet_manager.cc:1905] T af2f85c6a6e749cfb7b181d0aeab9c0d P 374ee6d3581f4e70829958bba908ead5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 19:04:38.499298 20542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:38.499181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45815 (local address 127.0.0.1:57034)
0929 19:04:38.499233 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0929 19:04:38.499275 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260929 19:04:38.499425 20070 catalog_manager.cc:4584] TS 97890b6ce5424f6d88e4c0ce32b40960 (127.0.98.1:45815): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet af2f85c6a6e749cfb7b181d0aeab9c0d: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45815: connect: Connection refused (error 111)
I20260929 19:04:38.503417 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUbnLHE/build/release/bin/kudu with pid 20406
I20260929 19:04:38.503429 20542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:38.503327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45873 (local address 127.0.0.1:43170)
0929 19:04:38.503383 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0929 19:04:38.503405 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:45873: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260929 19:04:38.507426 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUbnLHE/build/release/bin/kudu with pid 20053
2026-09-29T19:04:38Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (653 ms)
[----------] 6 tests from EncryptionPolicyTest (7971 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790708447806542-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 19:04:38 dist-test-slave-lqkv krb5kdc[20577](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 19:04:38 dist-test-slave-lqkv krb5kdc[20577](info): set up 2 sockets
Sep 29 19:04:38 dist-test-slave-lqkv krb5kdc[20577](info): commencing operation
krb5kdc: starting...
W20260929 19:04:39.460548 392 mini_kdc.cc:121] Time spent starting KDC: real 0.935s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790708447806542-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790708447806542-392-0/krb5kdc/test-user.keytab.
Sep 29 19:04:39 dist-test-slave-lqkv krb5kdc[20577](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708679, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T19:04:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T19:04:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790708447806542-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790708447806542-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 19:04:39.562855 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUbnLHE/build/release/bin/kudu
/tmp/dist-test-taskUbnLHE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790708447806542-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790708447806542-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790708447806542-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790708447806542-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41595
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790708447806542-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41595
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790708447806542-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790708447806542-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790708447806542-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:04:39.638032 20590 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:04:39.638206 20590 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:04:39.638226 20590 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:04:39.639309 20590 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 19:04:39.639350 20590 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:04:39.639364 20590 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:04:39.639400 20590 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 19:04:39.639429 20590 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 19:04:39.640785 20590 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790708447806542-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790708447806542-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41595
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790708447806542-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41595
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790708447806542-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20590
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790708447806542-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 18:09:13 UTC on ccycloud.quasar-hpkalf.root.comops.site
build id 4680913
I20260929 19:04:39.641038 20590 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:04:39.641244 20590 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 19:04:39.645632 20590 server_base.cc:1034] running on GCE node
W20260929 19:04:39.645579 20595 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:04:39.645565 20596 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:04:39.645586 20598 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:04:39.645992 20590 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:04:39.646255 20590 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:04:39.647401 20590 hybrid_clock.cc:648] HybridClock initialized: now 1790708679647388 us; error 27 us; skew 500 ppm
Sep 29 19:04:39 dist-test-slave-lqkv krb5kdc[20577](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708679, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:04:39.650355 20590 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 19:04:39.650727 20590 webserver.cc:460] Webserver started at http://127.0.0.1:42325/ using document root <none> and password file <none>
I20260929 19:04:39.650977 20590 fs_manager.cc:359] Metadata directory not provided
I20260929 19:04:39.651037 20590 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:04:39.651156 20590 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:04:39.652408 20590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790708447806542-392-0/minicluster-data/master-0/data/instance:
uuid: "fd75a9d271cd4f84b233e605fab94635"
format_stamp: "Formatted at 2026-09-29 19:04:39 on dist-test-slave-lqkv"
server_key: "5e97182a01340884196f87313ac14c46"
server_key_iv: "78d3b7a60a742509d5c511f4da9fed44"
server_key_version: "encryptionkey@0"
I20260929 19:04:39.652760 20590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790708447806542-392-0/minicluster-data/master-0/wal/instance:
uuid: "fd75a9d271cd4f84b233e605fab94635"
format_stamp: "Formatted at 2026-09-29 19:04:39 on dist-test-slave-lqkv"
server_key: "5e97182a01340884196f87313ac14c46"
server_key_iv: "78d3b7a60a742509d5c511f4da9fed44"
server_key_version: "encryptionkey@0"
I20260929 19:04:39.654234 20590 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260929 19:04:39.656418 20605 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:04:39.656563 20590 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.000s
I20260929 19:04:39.656636 20590 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790708447806542-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790708447806542-392-0/minicluster-data/master-0/wal
uuid: "fd75a9d271cd4f84b233e605fab94635"
format_stamp: "Formatted at 2026-09-29 19:04:39 on dist-test-slave-lqkv"
server_key: "5e97182a01340884196f87313ac14c46"
server_key_iv: "78d3b7a60a742509d5c511f4da9fed44"
server_key_version: "encryptionkey@0"
I20260929 19:04:39.656725 20590 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790708447806542-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790708447806542-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790708447806542-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:04:39.682379 20590 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:04:39.684897 20590 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:04:39.685029 20590 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:04:39.689037 20590 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41595
I20260929 19:04:39.689064 20657 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41595 every 8 connection(s)
I20260929 19:04:39.689369 20590 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790708447806542-392-0/minicluster-data/master-0/data/info.pb
I20260929 19:04:39.690096 20658 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:04:39.692600 20658 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd75a9d271cd4f84b233e605fab94635: Bootstrap starting.
I20260929 19:04:39.693179 20658 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fd75a9d271cd4f84b233e605fab94635: Neither blocks nor log segments found. Creating new log.
I20260929 19:04:39.693449 20658 log.cc:824] T 00000000000000000000000000000000 P fd75a9d271cd4f84b233e605fab94635: Log is configured to *not* fsync() on all Append() calls
I20260929 19:04:39.694074 20658 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd75a9d271cd4f84b233e605fab94635: No bootstrap required, opened a new log
I20260929 19:04:39.695590 20658 raft_consensus.cc:348] T 00000000000000000000000000000000 P fd75a9d271cd4f84b233e605fab94635 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd75a9d271cd4f84b233e605fab94635" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41595 } }
I20260929 19:04:39.695710 20658 raft_consensus.cc:374] T 00000000000000000000000000000000 P fd75a9d271cd4f84b233e605fab94635 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:04:39.695763 20658 raft_consensus.cc:729] T 00000000000000000000000000000000 P fd75a9d271cd4f84b233e605fab94635 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd75a9d271cd4f84b233e605fab94635, State: Initialized, Role: FOLLOWER
I20260929 19:04:39.695912 20658 consensus_queue.cc:260] T 00000000000000000000000000000000 P fd75a9d271cd4f84b233e605fab94635 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd75a9d271cd4f84b233e605fab94635" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41595 } }
I20260929 19:04:39.695988 20658 raft_consensus.cc:388] T 00000000000000000000000000000000 P fd75a9d271cd4f84b233e605fab94635 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 19:04:39.696033 20658 raft_consensus.cc:482] T 00000000000000000000000000000000 P fd75a9d271cd4f84b233e605fab94635 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 19:04:39.696076 20658 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fd75a9d271cd4f84b233e605fab94635 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:04:39.696530 20658 raft_consensus.cc:504] T 00000000000000000000000000000000 P fd75a9d271cd4f84b233e605fab94635 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd75a9d271cd4f84b233e605fab94635" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41595 } }
I20260929 19:04:39.696630 20658 leader_election.cc:302] T 00000000000000000000000000000000 P fd75a9d271cd4f84b233e605fab94635 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fd75a9d271cd4f84b233e605fab94635; no voters:
I20260929 19:04:39.696985 20658 leader_election.cc:288] T 00000000000000000000000000000000 P fd75a9d271cd4f84b233e605fab94635 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 19:04:39.697152 20662 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fd75a9d271cd4f84b233e605fab94635 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:04:39.697248 20658 sys_catalog.cc:564] T 00000000000000000000000000000000 P fd75a9d271cd4f84b233e605fab94635 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 19:04:39.697273 20662 raft_consensus.cc:686] T 00000000000000000000000000000000 P fd75a9d271cd4f84b233e605fab94635 [term 1 LEADER]: Becoming Leader. State: Replica: fd75a9d271cd4f84b233e605fab94635, State: Running, Role: LEADER
I20260929 19:04:39.697345 20662 consensus_queue.cc:237] T 00000000000000000000000000000000 P fd75a9d271cd4f84b233e605fab94635 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd75a9d271cd4f84b233e605fab94635" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41595 } }
I20260929 19:04:39.697706 20663 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd75a9d271cd4f84b233e605fab94635 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fd75a9d271cd4f84b233e605fab94635. Latest consensus state: current_term: 1 leader_uuid: "fd75a9d271cd4f84b233e605fab94635" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd75a9d271cd4f84b233e605fab94635" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41595 } } }
I20260929 19:04:39.697721 20661 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd75a9d271cd4f84b233e605fab94635 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fd75a9d271cd4f84b233e605fab94635" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd75a9d271cd4f84b233e605fab94635" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41595 } } }
I20260929 19:04:39.697944 20661 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd75a9d271cd4f84b233e605fab94635 [sys.catalog]: This master's current role is: LEADER
I20260929 19:04:39.697983 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUbnLHE/build/release/bin/kudu as pid 20590
I20260929 19:04:39.697892 20663 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd75a9d271cd4f84b233e605fab94635 [sys.catalog]: This master's current role is: LEADER
I20260929 19:04:39.698096 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790708447806542-392-0/minicluster-data/master-0/wal/instance
I20260929 19:04:39.698192 20667 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 19:04:39.698307 392 external_mini_cluster.cc:1428] Setting key 74bd32002b1e22ae3345ad1b10eb666c
I20260929 19:04:39.698680 20667 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 19:04:39.700172 20667 catalog_manager.cc:1347] Generated new cluster ID: e8575b51a5d4487d80d5d4c6c13ee91b
I20260929 19:04:39.700230 20667 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 29 19:04:39 dist-test-slave-lqkv krb5kdc[20577](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708679, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 19:04:39.709561 20680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:39.698908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44049 (local address 127.0.0.1:41595)
0929 19:04:39.699080 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0929 19:04:39.699090 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 19:04:39.699421 (+ 331us) server_negotiation.cc:407] Connection header received
0929 19:04:39.699535 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:04:39.699541 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:04:39.699591 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:04:39.699725 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:04:39.700573 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:39.701837 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:39.702831 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:39.703364 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:39.703448 (+ 84us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:04:39.705970 (+ 2522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:04:39.705974 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:04:39.705974 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:04:39.705986 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:04:39.707874 (+ 1888us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:04:39.708289 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:04:39.708291 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:04:39.708292 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:04:39.708404 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:04:39.708628 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:04:39.708628 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:04:39.708629 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:04:39.708977 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:04:39.709118 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0929 19:04:39.709222 (+ 104us) server_negotiation.cc:299] Negotiation successful
0929 19:04:39.709341 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":61,"threads_started":1}
I20260929 19:04:39.710333 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 19:04:39.719956 20667 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 19:04:39.720048 20680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:39.713109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47757 (local address 127.0.0.1:41595)
0929 19:04:39.713149 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0929 19:04:39.713151 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:04:39.713309 (+ 158us) server_negotiation.cc:407] Connection header received
0929 19:04:39.713356 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:04:39.713357 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:04:39.713365 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:04:39.713415 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:04:39.713987 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:39.714614 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:39.715507 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:39.715817 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:39.715880 (+ 63us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:04:39.717250 (+ 1370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:04:39.717251 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:04:39.717252 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:04:39.717258 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:04:39.718780 (+ 1522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:04:39.719198 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:04:39.719199 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:04:39.719199 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:04:39.719266 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:04:39.719509 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:04:39.719510 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:04:39.719510 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:04:39.719624 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:04:39.719740 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0929 19:04:39.719871 (+ 131us) server_negotiation.cc:299] Negotiation successful
0929 19:04:39.719940 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260929 19:04:39.720450 20667 catalog_manager.cc:1504] Loading token signing keys...
I20260929 19:04:39.728981 20667 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fd75a9d271cd4f84b233e605fab94635: Generated new TSK 0
I20260929 19:04:39.729142 20667 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 19:04:39.732029 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUbnLHE/build/release/bin/kudu with pid 20590
W20260929 19:04:39.737986 20575 connection.cc:535] client connection to 127.0.0.1:41595 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41595): Connection reset by peer (error 104)
2026-09-29T19:04:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1225 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790708447806542-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 19:04:39 dist-test-slave-lqkv krb5kdc[20689](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 19:04:39 dist-test-slave-lqkv krb5kdc[20689](info): set up 2 sockets
Sep 29 19:04:39 dist-test-slave-lqkv krb5kdc[20689](info): commencing operation
krb5kdc: starting...
W20260929 19:04:40.740490 392 mini_kdc.cc:121] Time spent starting KDC: real 0.989s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790708447806542-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790708447806542-392-0/krb5kdc/test-user.keytab.
Sep 29 19:04:40 dist-test-slave-lqkv krb5kdc[20689](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708680, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T19:04:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T19:04:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790708447806542-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790708447806542-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 19:04:40.843669 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUbnLHE/build/release/bin/kudu
/tmp/dist-test-taskUbnLHE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790708447806542-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790708447806542-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790708447806542-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790708447806542-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38675
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790708447806542-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38675
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790708447806542-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790708447806542-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790708447806542-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:04:40.915431 20702 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:04:40.915650 20702 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:04:40.915685 20702 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:04:40.916692 20702 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 19:04:40.916752 20702 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:04:40.916774 20702 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:04:40.916824 20702 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 19:04:40.916880 20702 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 19:04:40.918354 20702 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790708447806542-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790708447806542-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38675
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790708447806542-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38675
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790708447806542-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20702
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790708447806542-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 18:09:13 UTC on ccycloud.quasar-hpkalf.root.comops.site
build id 4680913
I20260929 19:04:40.918612 20702 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:04:40.918820 20702 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:04:40.923231 20710 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:04:40.923259 20708 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:04:40.923256 20707 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:04:40.923306 20702 server_base.cc:1034] running on GCE node
I20260929 19:04:40.923619 20702 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:04:40.923874 20702 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:04:40.924996 20702 hybrid_clock.cc:648] HybridClock initialized: now 1790708680924984 us; error 27 us; skew 500 ppm
Sep 29 19:04:40 dist-test-slave-lqkv krb5kdc[20689](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708680, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:04:40.931523 20702 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 19:04:40.931954 20702 webserver.cc:460] Webserver started at http://127.0.0.1:42619/ using document root <none> and password file <none>
I20260929 19:04:40.932112 20702 fs_manager.cc:359] Metadata directory not provided
I20260929 19:04:40.932148 20702 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:04:40.932226 20702 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:04:40.933220 20702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790708447806542-392-0/minicluster-data/master-0/data/instance:
uuid: "5bc1888742a54b359a33f2a7c023c8df"
format_stamp: "Formatted at 2026-09-29 19:04:40 on dist-test-slave-lqkv"
server_key: "a96d0cdf854122d2f5df039c1999ec84"
server_key_iv: "bdb87128b609d69b063dfdfe159cbe2c"
server_key_version: "encryptionkey@0"
I20260929 19:04:40.933506 20702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790708447806542-392-0/minicluster-data/master-0/wal/instance:
uuid: "5bc1888742a54b359a33f2a7c023c8df"
format_stamp: "Formatted at 2026-09-29 19:04:40 on dist-test-slave-lqkv"
server_key: "a96d0cdf854122d2f5df039c1999ec84"
server_key_iv: "bdb87128b609d69b063dfdfe159cbe2c"
server_key_version: "encryptionkey@0"
I20260929 19:04:40.934643 20702 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 19:04:40.935300 20717 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:04:40.935457 20702 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 19:04:40.935549 20702 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790708447806542-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790708447806542-392-0/minicluster-data/master-0/wal
uuid: "5bc1888742a54b359a33f2a7c023c8df"
format_stamp: "Formatted at 2026-09-29 19:04:40 on dist-test-slave-lqkv"
server_key: "a96d0cdf854122d2f5df039c1999ec84"
server_key_iv: "bdb87128b609d69b063dfdfe159cbe2c"
server_key_version: "encryptionkey@0"
I20260929 19:04:40.935626 20702 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790708447806542-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790708447806542-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790708447806542-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:04:40.946146 20702 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:04:40.947758 20702 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:04:40.947908 20702 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:04:40.951987 20702 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38675
I20260929 19:04:40.952030 20769 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38675 every 8 connection(s)
I20260929 19:04:40.952348 20702 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790708447806542-392-0/minicluster-data/master-0/data/info.pb
I20260929 19:04:40.953202 20770 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:04:40.955164 20770 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5bc1888742a54b359a33f2a7c023c8df: Bootstrap starting.
I20260929 19:04:40.955608 20770 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5bc1888742a54b359a33f2a7c023c8df: Neither blocks nor log segments found. Creating new log.
I20260929 19:04:40.955811 20770 log.cc:824] T 00000000000000000000000000000000 P 5bc1888742a54b359a33f2a7c023c8df: Log is configured to *not* fsync() on all Append() calls
I20260929 19:04:40.956346 20770 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5bc1888742a54b359a33f2a7c023c8df: No bootstrap required, opened a new log
I20260929 19:04:40.957273 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUbnLHE/build/release/bin/kudu as pid 20702
I20260929 19:04:40.957360 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790708447806542-392-0/minicluster-data/master-0/wal/instance
I20260929 19:04:40.957552 392 external_mini_cluster.cc:1428] Setting key 834726f5af6b08f8dff529b633b3c6ae
I20260929 19:04:40.958459 20770 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5bc1888742a54b359a33f2a7c023c8df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bc1888742a54b359a33f2a7c023c8df" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38675 } }
I20260929 19:04:40.958606 20770 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5bc1888742a54b359a33f2a7c023c8df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:04:40.958640 20770 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5bc1888742a54b359a33f2a7c023c8df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5bc1888742a54b359a33f2a7c023c8df, State: Initialized, Role: FOLLOWER
I20260929 19:04:40.958789 20770 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5bc1888742a54b359a33f2a7c023c8df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bc1888742a54b359a33f2a7c023c8df" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38675 } }
I20260929 19:04:40.958894 20770 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5bc1888742a54b359a33f2a7c023c8df [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 19:04:40.958927 20770 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5bc1888742a54b359a33f2a7c023c8df [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 19:04:40.958977 20770 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5bc1888742a54b359a33f2a7c023c8df [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:04:40.959640 20770 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5bc1888742a54b359a33f2a7c023c8df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bc1888742a54b359a33f2a7c023c8df" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38675 } }
I20260929 19:04:40.959731 20770 leader_election.cc:302] T 00000000000000000000000000000000 P 5bc1888742a54b359a33f2a7c023c8df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5bc1888742a54b359a33f2a7c023c8df; no voters:
I20260929 19:04:40.959854 20770 leader_election.cc:288] T 00000000000000000000000000000000 P 5bc1888742a54b359a33f2a7c023c8df [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 19:04:40.959913 20775 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5bc1888742a54b359a33f2a7c023c8df [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:04:40.960080 20770 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5bc1888742a54b359a33f2a7c023c8df [sys.catalog]: configured and running, proceeding with master startup.
I20260929 19:04:40.960098 20775 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5bc1888742a54b359a33f2a7c023c8df [term 1 LEADER]: Becoming Leader. State: Replica: 5bc1888742a54b359a33f2a7c023c8df, State: Running, Role: LEADER
I20260929 19:04:40.960177 20775 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5bc1888742a54b359a33f2a7c023c8df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bc1888742a54b359a33f2a7c023c8df" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38675 } }
I20260929 19:04:40.960494 20775 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5bc1888742a54b359a33f2a7c023c8df [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5bc1888742a54b359a33f2a7c023c8df. Latest consensus state: current_term: 1 leader_uuid: "5bc1888742a54b359a33f2a7c023c8df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bc1888742a54b359a33f2a7c023c8df" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38675 } } }
I20260929 19:04:40.960501 20776 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5bc1888742a54b359a33f2a7c023c8df [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5bc1888742a54b359a33f2a7c023c8df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bc1888742a54b359a33f2a7c023c8df" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38675 } } }
I20260929 19:04:40.960718 20776 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5bc1888742a54b359a33f2a7c023c8df [sys.catalog]: This master's current role is: LEADER
I20260929 19:04:40.960939 20775 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5bc1888742a54b359a33f2a7c023c8df [sys.catalog]: This master's current role is: LEADER
I20260929 19:04:40.961114 20784 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 19:04:40.961628 20784 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 19:04:40.963393 20784 catalog_manager.cc:1347] Generated new cluster ID: b27884f7d5164669883addd55c2a1b04
I20260929 19:04:40.963454 20784 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 19:04:40.963657 20774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:40.957989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38057 (local address 127.0.0.1:38675)
0929 19:04:40.958194 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0929 19:04:40.958203 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 19:04:40.958236 (+ 33us) server_negotiation.cc:407] Connection header received
0929 19:04:40.958423 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:04:40.958429 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:04:40.958478 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:04:40.958607 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:04:40.958863 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:40.960513 (+ 1650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:40.961692 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:40.962262 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:40.962388 (+ 126us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:04:40.963040 (+ 652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:04:40.963044 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:04:40.963044 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 19:04:40.963224 (+ 180us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:04:40.963280 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 19:04:40.963295 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:04:40.963377 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 19:04:40.963468 (+ 91us) server_negotiation.cc:299] Negotiation successful
0929 19:04:40.963493 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":97,"threads_started":1}
I20260929 19:04:40.964514 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 19:04:40.971359 20774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:40.965874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41149 (local address 127.0.0.1:38675)
0929 19:04:40.965907 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0929 19:04:40.965908 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 19:04:40.968954 (+ 3046us) server_negotiation.cc:407] Connection header received
0929 19:04:40.969017 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:04:40.969018 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:04:40.969027 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:04:40.969077 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:04:40.969210 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:40.969628 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:40.970331 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:40.970530 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:40.970597 (+ 67us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:04:40.971138 (+ 541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:04:40.971139 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:04:40.971139 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 19:04:40.971143 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:04:40.971153 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 19:04:40.971157 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:04:40.971204 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0929 19:04:40.971257 (+ 53us) server_negotiation.cc:299] Negotiation successful
0929 19:04:40.971266 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260929 19:04:40.974236 20784 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 19:04:40.974701 20784 catalog_manager.cc:1504] Loading token signing keys...
I20260929 19:04:40.979705 20784 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5bc1888742a54b359a33f2a7c023c8df: Generated new TSK 0
I20260929 19:04:40.979846 20784 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 19:04:40.987543 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUbnLHE/build/release/bin/kudu with pid 20702
W20260929 19:04:40.991223 20687 connection.cc:535] client connection to 127.0.0.1:38675 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38675): Connection reset by peer (error 104)
2026-09-29T19:04:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1253 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790708447806542-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 19:04:41 dist-test-slave-lqkv krb5kdc[20801](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 19:04:41 dist-test-slave-lqkv krb5kdc[20801](info): set up 2 sockets
Sep 29 19:04:41 dist-test-slave-lqkv krb5kdc[20801](info): commencing operation
krb5kdc: starting...
W20260929 19:04:41.960759 392 mini_kdc.cc:121] Time spent starting KDC: real 0.956s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790708447806542-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790708447806542-392-0/krb5kdc/test-user.keytab.
Sep 29 19:04:42 dist-test-slave-lqkv krb5kdc[20801](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708682, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T19:04:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T19:04:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790708447806542-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790708447806542-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 19:04:42.065230 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUbnLHE/build/release/bin/kudu
/tmp/dist-test-taskUbnLHE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790708447806542-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790708447806542-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790708447806542-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790708447806542-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43263
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790708447806542-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43263
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790708447806542-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790708447806542-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790708447806542-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:04:42.135635 20814 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:04:42.135843 20814 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:04:42.135880 20814 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:04:42.136967 20814 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 19:04:42.137061 20814 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:04:42.137087 20814 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:04:42.137120 20814 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 19:04:42.137144 20814 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 19:04:42.138509 20814 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790708447806542-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790708447806542-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43263
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790708447806542-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43263
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790708447806542-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20814
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790708447806542-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 18:09:13 UTC on ccycloud.quasar-hpkalf.root.comops.site
build id 4680913
I20260929 19:04:42.138731 20814 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:04:42.138911 20814 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:04:42.143240 20822 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:04:42.143312 20820 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:04:42.143451 20814 server_base.cc:1034] running on GCE node
W20260929 19:04:42.143229 20819 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:04:42.143708 20814 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:04:42.143956 20814 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:04:42.145080 20814 hybrid_clock.cc:648] HybridClock initialized: now 1790708682145069 us; error 25 us; skew 500 ppm
Sep 29 19:04:42 dist-test-slave-lqkv krb5kdc[20801](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708682, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:04:42.147104 20814 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 19:04:42.147528 20814 webserver.cc:460] Webserver started at http://127.0.0.1:33739/ using document root <none> and password file <none>
I20260929 19:04:42.147688 20814 fs_manager.cc:359] Metadata directory not provided
I20260929 19:04:42.147725 20814 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:04:42.147802 20814 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:04:42.148656 20814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790708447806542-392-0/minicluster-data/master-0/data/instance:
uuid: "284a6c5734b747469686b64152800e25"
format_stamp: "Formatted at 2026-09-29 19:04:42 on dist-test-slave-lqkv"
server_key: "16a795dcaf5d46a63a72c0b787e6da78"
server_key_iv: "85b24f0fdee8e8b616a3e244ebc4f5ba"
server_key_version: "encryptionkey@0"
I20260929 19:04:42.148955 20814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790708447806542-392-0/minicluster-data/master-0/wal/instance:
uuid: "284a6c5734b747469686b64152800e25"
format_stamp: "Formatted at 2026-09-29 19:04:42 on dist-test-slave-lqkv"
server_key: "16a795dcaf5d46a63a72c0b787e6da78"
server_key_iv: "85b24f0fdee8e8b616a3e244ebc4f5ba"
server_key_version: "encryptionkey@0"
I20260929 19:04:42.150105 20814 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260929 19:04:42.150709 20829 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:04:42.150878 20814 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 19:04:42.150952 20814 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790708447806542-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790708447806542-392-0/minicluster-data/master-0/wal
uuid: "284a6c5734b747469686b64152800e25"
format_stamp: "Formatted at 2026-09-29 19:04:42 on dist-test-slave-lqkv"
server_key: "16a795dcaf5d46a63a72c0b787e6da78"
server_key_iv: "85b24f0fdee8e8b616a3e244ebc4f5ba"
server_key_version: "encryptionkey@0"
I20260929 19:04:42.151000 20814 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790708447806542-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790708447806542-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790708447806542-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:04:42.153734 20814 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:04:42.155570 20814 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:04:42.155671 20814 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:04:42.159053 20814 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43263
I20260929 19:04:42.159112 20881 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43263 every 8 connection(s)
I20260929 19:04:42.159395 20814 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790708447806542-392-0/minicluster-data/master-0/data/info.pb
I20260929 19:04:42.160267 20882 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:04:42.162210 20882 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 284a6c5734b747469686b64152800e25: Bootstrap starting.
I20260929 19:04:42.162721 20882 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 284a6c5734b747469686b64152800e25: Neither blocks nor log segments found. Creating new log.
I20260929 19:04:42.162997 20882 log.cc:824] T 00000000000000000000000000000000 P 284a6c5734b747469686b64152800e25: Log is configured to *not* fsync() on all Append() calls
I20260929 19:04:42.163584 20882 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 284a6c5734b747469686b64152800e25: No bootstrap required, opened a new log
I20260929 19:04:42.165107 20882 raft_consensus.cc:348] T 00000000000000000000000000000000 P 284a6c5734b747469686b64152800e25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "284a6c5734b747469686b64152800e25" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43263 } }
I20260929 19:04:42.165335 20882 raft_consensus.cc:374] T 00000000000000000000000000000000 P 284a6c5734b747469686b64152800e25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:04:42.165390 20882 raft_consensus.cc:729] T 00000000000000000000000000000000 P 284a6c5734b747469686b64152800e25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 284a6c5734b747469686b64152800e25, State: Initialized, Role: FOLLOWER
I20260929 19:04:42.165525 20882 consensus_queue.cc:260] T 00000000000000000000000000000000 P 284a6c5734b747469686b64152800e25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "284a6c5734b747469686b64152800e25" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43263 } }
I20260929 19:04:42.165603 20882 raft_consensus.cc:388] T 00000000000000000000000000000000 P 284a6c5734b747469686b64152800e25 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 19:04:42.165643 20882 raft_consensus.cc:482] T 00000000000000000000000000000000 P 284a6c5734b747469686b64152800e25 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 19:04:42.165688 20882 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 284a6c5734b747469686b64152800e25 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:04:42.166149 20882 raft_consensus.cc:504] T 00000000000000000000000000000000 P 284a6c5734b747469686b64152800e25 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "284a6c5734b747469686b64152800e25" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43263 } }
I20260929 19:04:42.166249 20882 leader_election.cc:302] T 00000000000000000000000000000000 P 284a6c5734b747469686b64152800e25 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 284a6c5734b747469686b64152800e25; no voters:
I20260929 19:04:42.166395 20882 leader_election.cc:288] T 00000000000000000000000000000000 P 284a6c5734b747469686b64152800e25 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 19:04:42.166436 20885 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 284a6c5734b747469686b64152800e25 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:04:42.166630 20882 sys_catalog.cc:564] T 00000000000000000000000000000000 P 284a6c5734b747469686b64152800e25 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 19:04:42.166728 20885 raft_consensus.cc:686] T 00000000000000000000000000000000 P 284a6c5734b747469686b64152800e25 [term 1 LEADER]: Becoming Leader. State: Replica: 284a6c5734b747469686b64152800e25, State: Running, Role: LEADER
I20260929 19:04:42.166805 20885 consensus_queue.cc:237] T 00000000000000000000000000000000 P 284a6c5734b747469686b64152800e25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "284a6c5734b747469686b64152800e25" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43263 } }
I20260929 19:04:42.167187 20887 sys_catalog.cc:455] T 00000000000000000000000000000000 P 284a6c5734b747469686b64152800e25 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 284a6c5734b747469686b64152800e25. Latest consensus state: current_term: 1 leader_uuid: "284a6c5734b747469686b64152800e25" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "284a6c5734b747469686b64152800e25" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43263 } } }
I20260929 19:04:42.167292 20887 sys_catalog.cc:458] T 00000000000000000000000000000000 P 284a6c5734b747469686b64152800e25 [sys.catalog]: This master's current role is: LEADER
I20260929 19:04:42.167517 20894 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 19:04:42.167613 20886 sys_catalog.cc:455] T 00000000000000000000000000000000 P 284a6c5734b747469686b64152800e25 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "284a6c5734b747469686b64152800e25" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "284a6c5734b747469686b64152800e25" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43263 } } }
I20260929 19:04:42.167707 20886 sys_catalog.cc:458] T 00000000000000000000000000000000 P 284a6c5734b747469686b64152800e25 [sys.catalog]: This master's current role is: LEADER
I20260929 19:04:42.168141 20894 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 19:04:42.168751 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUbnLHE/build/release/bin/kudu as pid 20814
I20260929 19:04:42.168888 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790708447806542-392-0/minicluster-data/master-0/wal/instance
I20260929 19:04:42.169075 392 external_mini_cluster.cc:1428] Setting key 3c8dbff685776c8c1058ea9dadccf052
I20260929 19:04:42.173215 20894 catalog_manager.cc:1347] Generated new cluster ID: 4f7b906c40464f188162029e6496777b
I20260929 19:04:42.173264 20894 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 19:04:42.174700 20905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:42.169563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40963 (local address 127.0.0.1:43263)
0929 19:04:42.169713 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0929 19:04:42.169719 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 19:04:42.173016 (+ 3297us) server_negotiation.cc:407] Connection header received
0929 19:04:42.173163 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:04:42.173169 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:04:42.173204 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:04:42.173325 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:04:42.174300 (+ 975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:04:42.174303 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:04:42.174304 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 19:04:42.174307 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:04:42.174342 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 19:04:42.174359 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:04:42.174430 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 19:04:42.174496 (+ 66us) server_negotiation.cc:299] Negotiation successful
0929 19:04:42.174534 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":74,"threads_started":1}
I20260929 19:04:42.177614 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 19:04:42.183890 20894 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 19:04:42.184022 20894 catalog_manager.cc:1504] Loading token signing keys...
I20260929 19:04:42.185546 20905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:42.178885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53285 (local address 127.0.0.1:43263)
0929 19:04:42.178917 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0929 19:04:42.178919 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:04:42.183730 (+ 4811us) server_negotiation.cc:407] Connection header received
0929 19:04:42.183791 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:04:42.183791 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:04:42.183797 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:04:42.183845 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:04:42.185335 (+ 1490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:04:42.185337 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:04:42.185337 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 19:04:42.185341 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:04:42.185352 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 19:04:42.185356 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:04:42.185420 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0929 19:04:42.185465 (+ 45us) server_negotiation.cc:299] Negotiation successful
0929 19:04:42.185478 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260929 19:04:42.187621 20894 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 284a6c5734b747469686b64152800e25: Generated new TSK 0
I20260929 19:04:42.187763 20894 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 19:04:42.192166 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUbnLHE/build/release/bin/kudu with pid 20814
2026-09-29T19:04:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1204 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790708447806542-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 19:04:42 dist-test-slave-lqkv krb5kdc[20913](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 19:04:42 dist-test-slave-lqkv krb5kdc[20913](info): set up 2 sockets
Sep 29 19:04:42 dist-test-slave-lqkv krb5kdc[20913](info): commencing operation
krb5kdc: starting...
W20260929 19:04:43.168958 392 mini_kdc.cc:121] Time spent starting KDC: real 0.960s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790708447806542-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790708447806542-392-0/krb5kdc/test-user.keytab.
Sep 29 19:04:43 dist-test-slave-lqkv krb5kdc[20913](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708683, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T19:04:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T19:04:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790708447806542-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790708447806542-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 19:04:43.281744 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUbnLHE/build/release/bin/kudu
/tmp/dist-test-taskUbnLHE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790708447806542-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790708447806542-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790708447806542-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790708447806542-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40487
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790708447806542-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40487
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790708447806542-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790708447806542-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790708447806542-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:04:43.359437 20926 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:04:43.359627 20926 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:04:43.359647 20926 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:04:43.360729 20926 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 19:04:43.360767 20926 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:04:43.360780 20926 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:04:43.360841 20926 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 19:04:43.360867 20926 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 19:04:43.362283 20926 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790708447806542-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790708447806542-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40487
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790708447806542-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40487
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790708447806542-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20926
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790708447806542-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 18:09:13 UTC on ccycloud.quasar-hpkalf.root.comops.site
build id 4680913
I20260929 19:04:43.362516 20926 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:04:43.362726 20926 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:04:43.367465 20931 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:04:43.367443 20934 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:04:43.367444 20932 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:04:43.367470 20926 server_base.cc:1034] running on GCE node
I20260929 19:04:43.367873 20926 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:04:43.368177 20926 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:04:43.369338 20926 hybrid_clock.cc:648] HybridClock initialized: now 1790708683369326 us; error 31 us; skew 500 ppm
Sep 29 19:04:43 dist-test-slave-lqkv krb5kdc[20913](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708683, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:04:43.371634 20926 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 19:04:43.372066 20926 webserver.cc:460] Webserver started at http://127.0.0.1:40287/ using document root <none> and password file <none>
I20260929 19:04:43.372277 20926 fs_manager.cc:359] Metadata directory not provided
I20260929 19:04:43.372339 20926 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:04:43.372463 20926 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:04:43.373500 20926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790708447806542-392-0/minicluster-data/master-0/data/instance:
uuid: "d05ba557624e4d6790e9947fadff3b0a"
format_stamp: "Formatted at 2026-09-29 19:04:43 on dist-test-slave-lqkv"
server_key: "0ee0afa9200e7f3fc060ec620bad9455"
server_key_iv: "04b0e693a00c4a6ece7650159e1a2bf1"
server_key_version: "encryptionkey@0"
I20260929 19:04:43.373843 20926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790708447806542-392-0/minicluster-data/master-0/wal/instance:
uuid: "d05ba557624e4d6790e9947fadff3b0a"
format_stamp: "Formatted at 2026-09-29 19:04:43 on dist-test-slave-lqkv"
server_key: "0ee0afa9200e7f3fc060ec620bad9455"
server_key_iv: "04b0e693a00c4a6ece7650159e1a2bf1"
server_key_version: "encryptionkey@0"
I20260929 19:04:43.375109 20926 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 19:04:43.375891 20941 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:04:43.376120 20926 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 19:04:43.376228 20926 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790708447806542-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790708447806542-392-0/minicluster-data/master-0/wal
uuid: "d05ba557624e4d6790e9947fadff3b0a"
format_stamp: "Formatted at 2026-09-29 19:04:43 on dist-test-slave-lqkv"
server_key: "0ee0afa9200e7f3fc060ec620bad9455"
server_key_iv: "04b0e693a00c4a6ece7650159e1a2bf1"
server_key_version: "encryptionkey@0"
I20260929 19:04:43.376322 20926 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790708447806542-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790708447806542-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790708447806542-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:04:43.392782 20926 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:04:43.394560 20926 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:04:43.394717 20926 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:04:43.398593 20926 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40487
I20260929 19:04:43.398624 20993 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40487 every 8 connection(s)
I20260929 19:04:43.398957 20926 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790708447806542-392-0/minicluster-data/master-0/data/info.pb
I20260929 19:04:43.399742 20994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:04:43.402009 20994 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d05ba557624e4d6790e9947fadff3b0a: Bootstrap starting.
I20260929 19:04:43.402503 20994 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d05ba557624e4d6790e9947fadff3b0a: Neither blocks nor log segments found. Creating new log.
I20260929 19:04:43.402732 20994 log.cc:824] T 00000000000000000000000000000000 P d05ba557624e4d6790e9947fadff3b0a: Log is configured to *not* fsync() on all Append() calls
I20260929 19:04:43.403342 20994 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d05ba557624e4d6790e9947fadff3b0a: No bootstrap required, opened a new log
I20260929 19:04:43.404853 20994 raft_consensus.cc:348] T 00000000000000000000000000000000 P d05ba557624e4d6790e9947fadff3b0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d05ba557624e4d6790e9947fadff3b0a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40487 } }
I20260929 19:04:43.405006 20994 raft_consensus.cc:374] T 00000000000000000000000000000000 P d05ba557624e4d6790e9947fadff3b0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:04:43.405027 20994 raft_consensus.cc:729] T 00000000000000000000000000000000 P d05ba557624e4d6790e9947fadff3b0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d05ba557624e4d6790e9947fadff3b0a, State: Initialized, Role: FOLLOWER
I20260929 19:04:43.405122 20994 consensus_queue.cc:260] T 00000000000000000000000000000000 P d05ba557624e4d6790e9947fadff3b0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d05ba557624e4d6790e9947fadff3b0a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40487 } }
I20260929 19:04:43.405164 20994 raft_consensus.cc:388] T 00000000000000000000000000000000 P d05ba557624e4d6790e9947fadff3b0a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 19:04:43.405226 20994 raft_consensus.cc:482] T 00000000000000000000000000000000 P d05ba557624e4d6790e9947fadff3b0a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 19:04:43.405285 20994 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d05ba557624e4d6790e9947fadff3b0a [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:04:43.405800 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUbnLHE/build/release/bin/kudu as pid 20926
I20260929 19:04:43.405932 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790708447806542-392-0/minicluster-data/master-0/wal/instance
I20260929 19:04:43.405974 20994 raft_consensus.cc:504] T 00000000000000000000000000000000 P d05ba557624e4d6790e9947fadff3b0a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d05ba557624e4d6790e9947fadff3b0a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40487 } }
I20260929 19:04:43.406082 20994 leader_election.cc:302] T 00000000000000000000000000000000 P d05ba557624e4d6790e9947fadff3b0a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d05ba557624e4d6790e9947fadff3b0a; no voters:
I20260929 19:04:43.406128 392 external_mini_cluster.cc:1428] Setting key 24ca85830a245515ea4ac6482187be7f
I20260929 19:04:43.406252 20994 leader_election.cc:288] T 00000000000000000000000000000000 P d05ba557624e4d6790e9947fadff3b0a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 19:04:43.406318 20997 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d05ba557624e4d6790e9947fadff3b0a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:04:43.406476 20997 raft_consensus.cc:686] T 00000000000000000000000000000000 P d05ba557624e4d6790e9947fadff3b0a [term 1 LEADER]: Becoming Leader. State: Replica: d05ba557624e4d6790e9947fadff3b0a, State: Running, Role: LEADER
I20260929 19:04:43.406507 20994 sys_catalog.cc:564] T 00000000000000000000000000000000 P d05ba557624e4d6790e9947fadff3b0a [sys.catalog]: configured and running, proceeding with master startup.
I20260929 19:04:43.406642 20997 consensus_queue.cc:237] T 00000000000000000000000000000000 P d05ba557624e4d6790e9947fadff3b0a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d05ba557624e4d6790e9947fadff3b0a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40487 } }
I20260929 19:04:43.407024 20998 sys_catalog.cc:455] T 00000000000000000000000000000000 P d05ba557624e4d6790e9947fadff3b0a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d05ba557624e4d6790e9947fadff3b0a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d05ba557624e4d6790e9947fadff3b0a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40487 } } }
I20260929 19:04:43.407037 20999 sys_catalog.cc:455] T 00000000000000000000000000000000 P d05ba557624e4d6790e9947fadff3b0a [sys.catalog]: SysCatalogTable state changed. Reason: New leader d05ba557624e4d6790e9947fadff3b0a. Latest consensus state: current_term: 1 leader_uuid: "d05ba557624e4d6790e9947fadff3b0a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d05ba557624e4d6790e9947fadff3b0a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40487 } } }
I20260929 19:04:43.407162 20998 sys_catalog.cc:458] T 00000000000000000000000000000000 P d05ba557624e4d6790e9947fadff3b0a [sys.catalog]: This master's current role is: LEADER
I20260929 19:04:43.407189 20999 sys_catalog.cc:458] T 00000000000000000000000000000000 P d05ba557624e4d6790e9947fadff3b0a [sys.catalog]: This master's current role is: LEADER
I20260929 19:04:43.407526 21003 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 19:04:43.408020 21003 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 19:04:43.410900 21003 catalog_manager.cc:1347] Generated new cluster ID: 0a5015da5400462d9a5f9fde44225aa8
I20260929 19:04:43.410962 21003 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 29 19:04:43 dist-test-slave-lqkv krb5kdc[20913](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708683, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 19:04:43.418512 21001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:43.406736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37155 (local address 127.0.0.1:40487)
0929 19:04:43.408404 (+ 1668us) server_negotiation.cc:206] Beginning negotiation
0929 19:04:43.408414 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 19:04:43.408448 (+ 34us) server_negotiation.cc:407] Connection header received
0929 19:04:43.408550 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:04:43.408555 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:04:43.408638 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:04:43.408775 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:04:43.409562 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:43.410442 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:43.411514 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:43.412119 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:43.412212 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:04:43.414909 (+ 2697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:04:43.414914 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:04:43.414914 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:04:43.414925 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:04:43.416852 (+ 1927us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:04:43.417223 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:04:43.417226 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:04:43.417226 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:04:43.417339 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:04:43.417543 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:04:43.417544 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:04:43.417544 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:04:43.417835 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:04:43.417992 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0929 19:04:43.418067 (+ 75us) server_negotiation.cc:299] Negotiation successful
0929 19:04:43.418304 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1622,"thread_start_us":68,"threads_started":1}
I20260929 19:04:43.419371 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 19:04:43.428092 21001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:43.421043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43391 (local address 127.0.0.1:40487)
0929 19:04:43.421075 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0929 19:04:43.421076 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 19:04:43.421186 (+ 110us) server_negotiation.cc:407] Connection header received
0929 19:04:43.421237 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:04:43.421237 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:04:43.421255 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:04:43.421310 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:04:43.421891 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:43.422590 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:43.423595 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:43.423885 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:43.423939 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:04:43.425329 (+ 1390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:04:43.425330 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:04:43.425330 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:04:43.425337 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:04:43.426817 (+ 1480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:04:43.427207 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:04:43.427208 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:04:43.427208 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:04:43.427269 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:04:43.427511 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:04:43.427512 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:04:43.427512 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:04:43.427670 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:04:43.427776 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0929 19:04:43.427905 (+ 129us) server_negotiation.cc:299] Negotiation successful
0929 19:04:43.427996 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260929 19:04:43.430161 21003 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 19:04:43.430615 21003 catalog_manager.cc:1504] Loading token signing keys...
I20260929 19:04:43.438381 21003 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d05ba557624e4d6790e9947fadff3b0a: Generated new TSK 0
I20260929 19:04:43.438565 21003 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 19:04:43.443349 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUbnLHE/build/release/bin/kudu with pid 20926
2026-09-29T19:04:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1251 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790708447806542-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 19:04:43 dist-test-slave-lqkv krb5kdc[21025](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 19:04:43 dist-test-slave-lqkv krb5kdc[21025](info): set up 2 sockets
Sep 29 19:04:43 dist-test-slave-lqkv krb5kdc[21025](info): commencing operation
krb5kdc: starting...
W20260929 19:04:44.452953 392 mini_kdc.cc:121] Time spent starting KDC: real 0.992s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790708447806542-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790708447806542-392-0/krb5kdc/test-user.keytab.
Sep 29 19:04:44 dist-test-slave-lqkv krb5kdc[21025](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708684, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T19:04:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T19:04:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790708447806542-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790708447806542-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 19:04:44.557642 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUbnLHE/build/release/bin/kudu
/tmp/dist-test-taskUbnLHE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790708447806542-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790708447806542-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790708447806542-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790708447806542-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41789
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790708447806542-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41789
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790708447806542-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790708447806542-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790708447806542-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:04:44.648386 21038 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:04:44.648571 21038 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:04:44.648591 21038 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:04:44.649681 21038 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 19:04:44.649717 21038 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:04:44.649729 21038 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:04:44.649762 21038 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 19:04:44.649786 21038 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 19:04:44.651007 21038 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790708447806542-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790708447806542-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41789
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790708447806542-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41789
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790708447806542-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21038
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790708447806542-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 18:09:13 UTC on ccycloud.quasar-hpkalf.root.comops.site
build id 4680913
I20260929 19:04:44.651225 21038 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:04:44.651408 21038 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:04:44.655572 21043 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:04:44.655572 21044 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:04:44.655627 21046 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:04:44.655673 21038 server_base.cc:1034] running on GCE node
I20260929 19:04:44.655975 21038 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:04:44.656236 21038 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:04:44.657383 21038 hybrid_clock.cc:648] HybridClock initialized: now 1790708684657364 us; error 27 us; skew 500 ppm
Sep 29 19:04:44 dist-test-slave-lqkv krb5kdc[21025](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708684, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:04:44.659551 21038 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 19:04:44.659956 21038 webserver.cc:460] Webserver started at http://127.0.0.1:40283/ using document root <none> and password file <none>
I20260929 19:04:44.660146 21038 fs_manager.cc:359] Metadata directory not provided
I20260929 19:04:44.660205 21038 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:04:44.660337 21038 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:04:44.661327 21038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790708447806542-392-0/minicluster-data/master-0/data/instance:
uuid: "0077e6ac9cf246b098ffee9608fb4d7e"
format_stamp: "Formatted at 2026-09-29 19:04:44 on dist-test-slave-lqkv"
server_key: "c3094f9c41575cbbb097932e875d92d4"
server_key_iv: "f850f1634d8c65adbdb8aae806d43bf0"
server_key_version: "encryptionkey@0"
I20260929 19:04:44.661696 21038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790708447806542-392-0/minicluster-data/master-0/wal/instance:
uuid: "0077e6ac9cf246b098ffee9608fb4d7e"
format_stamp: "Formatted at 2026-09-29 19:04:44 on dist-test-slave-lqkv"
server_key: "c3094f9c41575cbbb097932e875d92d4"
server_key_iv: "f850f1634d8c65adbdb8aae806d43bf0"
server_key_version: "encryptionkey@0"
I20260929 19:04:44.662834 21038 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 19:04:44.663475 21053 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:04:44.663620 21038 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 19:04:44.663708 21038 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790708447806542-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790708447806542-392-0/minicluster-data/master-0/wal
uuid: "0077e6ac9cf246b098ffee9608fb4d7e"
format_stamp: "Formatted at 2026-09-29 19:04:44 on dist-test-slave-lqkv"
server_key: "c3094f9c41575cbbb097932e875d92d4"
server_key_iv: "f850f1634d8c65adbdb8aae806d43bf0"
server_key_version: "encryptionkey@0"
I20260929 19:04:44.663777 21038 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790708447806542-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790708447806542-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790708447806542-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:04:44.683743 21038 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:04:44.685412 21038 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:04:44.685564 21038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:04:44.689365 21038 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41789
I20260929 19:04:44.689407 21105 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41789 every 8 connection(s)
I20260929 19:04:44.689738 21038 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790708447806542-392-0/minicluster-data/master-0/data/info.pb
I20260929 19:04:44.690480 21106 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:04:44.691527 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUbnLHE/build/release/bin/kudu as pid 21038
I20260929 19:04:44.691649 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790708447806542-392-0/minicluster-data/master-0/wal/instance
I20260929 19:04:44.691833 392 external_mini_cluster.cc:1428] Setting key e92365b66b7d76919abdb904ad77b8fe
I20260929 19:04:44.692950 21106 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0077e6ac9cf246b098ffee9608fb4d7e: Bootstrap starting.
I20260929 19:04:44.693565 21106 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0077e6ac9cf246b098ffee9608fb4d7e: Neither blocks nor log segments found. Creating new log.
I20260929 19:04:44.693837 21106 log.cc:824] T 00000000000000000000000000000000 P 0077e6ac9cf246b098ffee9608fb4d7e: Log is configured to *not* fsync() on all Append() calls
I20260929 19:04:44.694509 21106 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0077e6ac9cf246b098ffee9608fb4d7e: No bootstrap required, opened a new log
I20260929 19:04:44.696472 21106 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0077e6ac9cf246b098ffee9608fb4d7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0077e6ac9cf246b098ffee9608fb4d7e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41789 } }
I20260929 19:04:44.696630 21106 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0077e6ac9cf246b098ffee9608fb4d7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:04:44.696666 21106 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0077e6ac9cf246b098ffee9608fb4d7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0077e6ac9cf246b098ffee9608fb4d7e, State: Initialized, Role: FOLLOWER
I20260929 19:04:44.696774 21106 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0077e6ac9cf246b098ffee9608fb4d7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0077e6ac9cf246b098ffee9608fb4d7e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41789 } }
I20260929 19:04:44.696864 21106 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0077e6ac9cf246b098ffee9608fb4d7e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 19:04:44.696938 21106 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0077e6ac9cf246b098ffee9608fb4d7e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 19:04:44.696993 21106 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0077e6ac9cf246b098ffee9608fb4d7e [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:04:44.697069 21109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:44.692307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38507 (local address 127.0.0.1:41789)
0929 19:04:44.692476 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0929 19:04:44.692484 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 19:04:44.692515 (+ 31us) server_negotiation.cc:407] Connection header received
0929 19:04:44.692952 (+ 437us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:04:44.692959 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:04:44.693021 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:04:44.693148 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:04:44.693377 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:44.694314 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:44.695303 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:44.695664 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:44.695730 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:04:44.696306 (+ 576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:04:44.696309 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:04:44.696309 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 19:04:44.696445 (+ 136us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:04:44.696467 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 19:04:44.696476 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:04:44.696519 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0929 19:04:44.696610 (+ 91us) server_negotiation.cc:299] Negotiation successful
0929 19:04:44.696771 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":83,"threads_started":1}
I20260929 19:04:44.697598 21106 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0077e6ac9cf246b098ffee9608fb4d7e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0077e6ac9cf246b098ffee9608fb4d7e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41789 } }
I20260929 19:04:44.697701 21106 leader_election.cc:302] T 00000000000000000000000000000000 P 0077e6ac9cf246b098ffee9608fb4d7e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0077e6ac9cf246b098ffee9608fb4d7e; no voters:
I20260929 19:04:44.697882 21106 leader_election.cc:288] T 00000000000000000000000000000000 P 0077e6ac9cf246b098ffee9608fb4d7e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 19:04:44.697882 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 19:04:44.698037 21112 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0077e6ac9cf246b098ffee9608fb4d7e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:04:44.698102 21106 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0077e6ac9cf246b098ffee9608fb4d7e [sys.catalog]: configured and running, proceeding with master startup.
I20260929 19:04:44.698194 21112 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0077e6ac9cf246b098ffee9608fb4d7e [term 1 LEADER]: Becoming Leader. State: Replica: 0077e6ac9cf246b098ffee9608fb4d7e, State: Running, Role: LEADER
I20260929 19:04:44.698342 21112 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0077e6ac9cf246b098ffee9608fb4d7e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0077e6ac9cf246b098ffee9608fb4d7e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41789 } }
I20260929 19:04:44.698697 21112 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0077e6ac9cf246b098ffee9608fb4d7e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0077e6ac9cf246b098ffee9608fb4d7e. Latest consensus state: current_term: 1 leader_uuid: "0077e6ac9cf246b098ffee9608fb4d7e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0077e6ac9cf246b098ffee9608fb4d7e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41789 } } }
I20260929 19:04:44.698704 21113 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0077e6ac9cf246b098ffee9608fb4d7e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0077e6ac9cf246b098ffee9608fb4d7e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0077e6ac9cf246b098ffee9608fb4d7e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41789 } } }
I20260929 19:04:44.698827 21113 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0077e6ac9cf246b098ffee9608fb4d7e [sys.catalog]: This master's current role is: LEADER
I20260929 19:04:44.699024 21112 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0077e6ac9cf246b098ffee9608fb4d7e [sys.catalog]: This master's current role is: LEADER
W20260929 19:04:44.699826 21131 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0077e6ac9cf246b098ffee9608fb4d7e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 19:04:44.699904 21131 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 19:04:44.699945 21122 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 19:04:44.700088 21122 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 19:04:44.701788 21122 catalog_manager.cc:1347] Generated new cluster ID: 1945da2a959f4c249bdb1ed39467f603
I20260929 19:04:44.701846 21122 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 19:04:44.702540 21109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:44.699317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33805 (local address 127.0.0.1:41789)
0929 19:04:44.699350 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0929 19:04:44.699351 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 19:04:44.699614 (+ 263us) server_negotiation.cc:407] Connection header received
0929 19:04:44.699703 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:04:44.699703 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:04:44.699712 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:04:44.699761 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:04:44.699890 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:44.700430 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:44.701352 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:44.701618 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:44.701717 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:04:44.702311 (+ 594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:04:44.702312 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:04:44.702313 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 19:04:44.702316 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:04:44.702327 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 19:04:44.702331 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:04:44.702370 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0929 19:04:44.702431 (+ 61us) server_negotiation.cc:299] Negotiation successful
0929 19:04:44.702451 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260929 19:04:44.729310 21122 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 19:04:44.729925 21122 catalog_manager.cc:1504] Loading token signing keys...
I20260929 19:04:44.737058 21122 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0077e6ac9cf246b098ffee9608fb4d7e: Generated new TSK 0
I20260929 19:04:44.737210 21122 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 19:04:44.740926 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUbnLHE/build/release/bin/kudu with pid 21038
2026-09-29T19:04:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1297 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790708447806542-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 19:04:44 dist-test-slave-lqkv krb5kdc[21137](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 19:04:44 dist-test-slave-lqkv krb5kdc[21137](info): set up 2 sockets
Sep 29 19:04:44 dist-test-slave-lqkv krb5kdc[21137](info): commencing operation
krb5kdc: starting...
W20260929 19:04:45.716337 392 mini_kdc.cc:121] Time spent starting KDC: real 0.958s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790708447806542-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790708447806542-392-0/krb5kdc/test-user.keytab.
Sep 29 19:04:45 dist-test-slave-lqkv krb5kdc[21137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708685, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T19:04:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T19:04:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790708447806542-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790708447806542-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 19:04:45.819882 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUbnLHE/build/release/bin/kudu
/tmp/dist-test-taskUbnLHE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790708447806542-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790708447806542-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790708447806542-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790708447806542-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36155
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790708447806542-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36155
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790708447806542-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790708447806542-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790708447806542-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:04:45.893131 21150 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:04:45.893286 21150 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:04:45.893306 21150 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:04:45.894316 21150 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 19:04:45.894353 21150 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:04:45.894366 21150 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:04:45.894399 21150 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 19:04:45.894424 21150 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 19:04:45.895684 21150 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790708447806542-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790708447806542-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36155
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790708447806542-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36155
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790708447806542-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21150
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790708447806542-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 18:09:13 UTC on ccycloud.quasar-hpkalf.root.comops.site
build id 4680913
I20260929 19:04:45.895916 21150 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:04:45.896116 21150 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:04:45.900441 21156 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:04:45.900471 21158 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:04:45.900471 21155 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:04:45.900714 21150 server_base.cc:1034] running on GCE node
I20260929 19:04:45.900954 21150 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:04:45.901206 21150 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:04:45.902343 21150 hybrid_clock.cc:648] HybridClock initialized: now 1790708685902331 us; error 25 us; skew 500 ppm
Sep 29 19:04:45 dist-test-slave-lqkv krb5kdc[21137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708685, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:04:45.904476 21150 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 19:04:45.904953 21150 webserver.cc:460] Webserver started at http://127.0.0.1:42043/ using document root <none> and password file <none>
I20260929 19:04:45.905166 21150 fs_manager.cc:359] Metadata directory not provided
I20260929 19:04:45.905223 21150 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:04:45.905332 21150 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:04:45.906200 21150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790708447806542-392-0/minicluster-data/master-0/data/instance:
uuid: "a310d232451d4ebba592e004cf3ca0a7"
format_stamp: "Formatted at 2026-09-29 19:04:45 on dist-test-slave-lqkv"
server_key: "eb24096db7980ca27eb73ea36f2dd9dc"
server_key_iv: "a49551240a9c24c150e9124aa4553824"
server_key_version: "encryptionkey@0"
I20260929 19:04:45.906515 21150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790708447806542-392-0/minicluster-data/master-0/wal/instance:
uuid: "a310d232451d4ebba592e004cf3ca0a7"
format_stamp: "Formatted at 2026-09-29 19:04:45 on dist-test-slave-lqkv"
server_key: "eb24096db7980ca27eb73ea36f2dd9dc"
server_key_iv: "a49551240a9c24c150e9124aa4553824"
server_key_version: "encryptionkey@0"
I20260929 19:04:45.907693 21150 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 19:04:45.908337 21165 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:04:45.908527 21150 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 19:04:45.908612 21150 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790708447806542-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790708447806542-392-0/minicluster-data/master-0/wal
uuid: "a310d232451d4ebba592e004cf3ca0a7"
format_stamp: "Formatted at 2026-09-29 19:04:45 on dist-test-slave-lqkv"
server_key: "eb24096db7980ca27eb73ea36f2dd9dc"
server_key_iv: "a49551240a9c24c150e9124aa4553824"
server_key_version: "encryptionkey@0"
I20260929 19:04:45.908689 21150 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790708447806542-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790708447806542-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790708447806542-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:04:45.911450 21150 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:04:45.913013 21150 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:04:45.913162 21150 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:04:45.916512 21150 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36155
I20260929 19:04:45.916538 21217 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36155 every 8 connection(s)
I20260929 19:04:45.916837 21150 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790708447806542-392-0/minicluster-data/master-0/data/info.pb
I20260929 19:04:45.917695 21218 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:04:45.919655 21218 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a310d232451d4ebba592e004cf3ca0a7: Bootstrap starting.
I20260929 19:04:45.920141 21218 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a310d232451d4ebba592e004cf3ca0a7: Neither blocks nor log segments found. Creating new log.
I20260929 19:04:45.920385 21218 log.cc:824] T 00000000000000000000000000000000 P a310d232451d4ebba592e004cf3ca0a7: Log is configured to *not* fsync() on all Append() calls
I20260929 19:04:45.921002 21218 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a310d232451d4ebba592e004cf3ca0a7: No bootstrap required, opened a new log
I20260929 19:04:45.922327 21218 raft_consensus.cc:348] T 00000000000000000000000000000000 P a310d232451d4ebba592e004cf3ca0a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a310d232451d4ebba592e004cf3ca0a7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36155 } }
I20260929 19:04:45.922438 21218 raft_consensus.cc:374] T 00000000000000000000000000000000 P a310d232451d4ebba592e004cf3ca0a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:04:45.922497 21218 raft_consensus.cc:729] T 00000000000000000000000000000000 P a310d232451d4ebba592e004cf3ca0a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a310d232451d4ebba592e004cf3ca0a7, State: Initialized, Role: FOLLOWER
I20260929 19:04:45.922639 21218 consensus_queue.cc:260] T 00000000000000000000000000000000 P a310d232451d4ebba592e004cf3ca0a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a310d232451d4ebba592e004cf3ca0a7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36155 } }
I20260929 19:04:45.922709 21218 raft_consensus.cc:388] T 00000000000000000000000000000000 P a310d232451d4ebba592e004cf3ca0a7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 19:04:45.922747 21218 raft_consensus.cc:482] T 00000000000000000000000000000000 P a310d232451d4ebba592e004cf3ca0a7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 19:04:45.922789 21218 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a310d232451d4ebba592e004cf3ca0a7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:04:45.923282 21218 raft_consensus.cc:504] T 00000000000000000000000000000000 P a310d232451d4ebba592e004cf3ca0a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a310d232451d4ebba592e004cf3ca0a7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36155 } }
I20260929 19:04:45.923302 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUbnLHE/build/release/bin/kudu as pid 21150
I20260929 19:04:45.923383 21218 leader_election.cc:302] T 00000000000000000000000000000000 P a310d232451d4ebba592e004cf3ca0a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a310d232451d4ebba592e004cf3ca0a7; no voters:
I20260929 19:04:45.923416 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790708447806542-392-0/minicluster-data/master-0/wal/instance
I20260929 19:04:45.923547 21218 leader_election.cc:288] T 00000000000000000000000000000000 P a310d232451d4ebba592e004cf3ca0a7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 19:04:45.923568 21221 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a310d232451d4ebba592e004cf3ca0a7 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:04:45.923573 392 external_mini_cluster.cc:1428] Setting key c10e23479db22688549d14894507f3f6
I20260929 19:04:45.923674 21221 raft_consensus.cc:686] T 00000000000000000000000000000000 P a310d232451d4ebba592e004cf3ca0a7 [term 1 LEADER]: Becoming Leader. State: Replica: a310d232451d4ebba592e004cf3ca0a7, State: Running, Role: LEADER
I20260929 19:04:45.923744 21221 consensus_queue.cc:237] T 00000000000000000000000000000000 P a310d232451d4ebba592e004cf3ca0a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a310d232451d4ebba592e004cf3ca0a7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36155 } }
I20260929 19:04:45.923769 21218 sys_catalog.cc:564] T 00000000000000000000000000000000 P a310d232451d4ebba592e004cf3ca0a7 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 19:04:45.924134 21222 sys_catalog.cc:455] T 00000000000000000000000000000000 P a310d232451d4ebba592e004cf3ca0a7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a310d232451d4ebba592e004cf3ca0a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a310d232451d4ebba592e004cf3ca0a7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36155 } } }
I20260929 19:04:45.924220 21222 sys_catalog.cc:458] T 00000000000000000000000000000000 P a310d232451d4ebba592e004cf3ca0a7 [sys.catalog]: This master's current role is: LEADER
I20260929 19:04:45.924453 21223 sys_catalog.cc:455] T 00000000000000000000000000000000 P a310d232451d4ebba592e004cf3ca0a7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a310d232451d4ebba592e004cf3ca0a7. Latest consensus state: current_term: 1 leader_uuid: "a310d232451d4ebba592e004cf3ca0a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a310d232451d4ebba592e004cf3ca0a7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36155 } } }
I20260929 19:04:45.924518 21223 sys_catalog.cc:458] T 00000000000000000000000000000000 P a310d232451d4ebba592e004cf3ca0a7 [sys.catalog]: This master's current role is: LEADER
I20260929 19:04:45.924633 21229 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 19:04:45.925164 21229 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 19:04:45.926101 21227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:45.924148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34427 (local address 127.0.0.1:36155)
0929 19:04:45.924520 (+ 372us) server_negotiation.cc:206] Beginning negotiation
0929 19:04:45.924528 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 19:04:45.924612 (+ 84us) server_negotiation.cc:407] Connection header received
0929 19:04:45.924727 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:04:45.924734 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:04:45.924762 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:04:45.924912 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:04:45.925553 (+ 641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:04:45.925556 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:04:45.925557 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 19:04:45.925561 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:04:45.925601 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 19:04:45.925615 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:04:45.925877 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0929 19:04:45.925906 (+ 29us) server_negotiation.cc:299] Negotiation successful
0929 19:04:45.925941 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":116,"threads_started":1}
I20260929 19:04:45.926966 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 19:04:45.927057 21229 catalog_manager.cc:1347] Generated new cluster ID: fa3c4eeb39224e0db65a2913713f2e87
I20260929 19:04:45.927093 21229 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 19:04:45.929454 21227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:45.928380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47389 (local address 127.0.0.1:36155)
0929 19:04:45.928409 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0929 19:04:45.928410 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 19:04:45.928518 (+ 108us) server_negotiation.cc:407] Connection header received
0929 19:04:45.928611 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:04:45.928612 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:04:45.928616 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:04:45.928640 (+ 24us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:04:45.929249 (+ 609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:04:45.929250 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:04:45.929251 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 19:04:45.929254 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:04:45.929264 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 19:04:45.929267 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:04:45.929292 (+ 25us) server_negotiation.cc:1035] Waiting for connection context
0929 19:04:45.929374 (+ 82us) server_negotiation.cc:299] Negotiation successful
0929 19:04:45.929384 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260929 19:04:45.933914 21229 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 19:04:45.934038 21229 catalog_manager.cc:1504] Loading token signing keys...
I20260929 19:04:45.943825 21229 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a310d232451d4ebba592e004cf3ca0a7: Generated new TSK 0
I20260929 19:04:45.943953 21229 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 19:04:45.953305 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUbnLHE/build/release/bin/kudu with pid 21150
2026-09-29T19:04:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1212 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790708447806542-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 19:04:45 dist-test-slave-lqkv krb5kdc[21249](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 19:04:45 dist-test-slave-lqkv krb5kdc[21249](info): set up 2 sockets
Sep 29 19:04:45 dist-test-slave-lqkv krb5kdc[21249](info): commencing operation
krb5kdc: starting...
W20260929 19:04:46.916260 392 mini_kdc.cc:121] Time spent starting KDC: real 0.946s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790708447806542-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790708447806542-392-0/krb5kdc/test-user.keytab.
Sep 29 19:04:46 dist-test-slave-lqkv krb5kdc[21249](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708686, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T19:04:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T19:04:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790708447806542-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790708447806542-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 19:04:47.019098 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUbnLHE/build/release/bin/kudu
/tmp/dist-test-taskUbnLHE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790708447806542-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790708447806542-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790708447806542-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790708447806542-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45893
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790708447806542-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45893
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790708447806542-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790708447806542-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790708447806542-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:04:47.092350 21262 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:04:47.092541 21262 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:04:47.092576 21262 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:04:47.094025 21262 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 19:04:47.094089 21262 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:04:47.094113 21262 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:04:47.094133 21262 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 19:04:47.094158 21262 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 19:04:47.095566 21262 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790708447806542-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790708447806542-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45893
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790708447806542-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45893
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790708447806542-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21262
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790708447806542-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 18:09:13 UTC on ccycloud.quasar-hpkalf.root.comops.site
build id 4680913
I20260929 19:04:47.095791 21262 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:04:47.095999 21262 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:04:47.100481 21267 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:04:47.100530 21270 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:04:47.100559 21268 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:04:47.100648 21262 server_base.cc:1034] running on GCE node
I20260929 19:04:47.100977 21262 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:04:47.101238 21262 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:04:47.102368 21262 hybrid_clock.cc:648] HybridClock initialized: now 1790708687102355 us; error 28 us; skew 500 ppm
Sep 29 19:04:47 dist-test-slave-lqkv krb5kdc[21249](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708687, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:04:47.104417 21262 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 19:04:47.104789 21262 webserver.cc:460] Webserver started at http://127.0.0.1:43239/ using document root <none> and password file <none>
I20260929 19:04:47.105036 21262 fs_manager.cc:359] Metadata directory not provided
I20260929 19:04:47.105075 21262 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:04:47.105197 21262 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:04:47.106083 21262 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790708447806542-392-0/minicluster-data/master-0/data/instance:
uuid: "0afc723a71b34f758453772cd66fd5d2"
format_stamp: "Formatted at 2026-09-29 19:04:47 on dist-test-slave-lqkv"
server_key: "ff0bc3b3b15ef0353a88b96173abb092"
server_key_iv: "00c01259669d543da35698bfaea0286b"
server_key_version: "encryptionkey@0"
I20260929 19:04:47.106396 21262 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790708447806542-392-0/minicluster-data/master-0/wal/instance:
uuid: "0afc723a71b34f758453772cd66fd5d2"
format_stamp: "Formatted at 2026-09-29 19:04:47 on dist-test-slave-lqkv"
server_key: "ff0bc3b3b15ef0353a88b96173abb092"
server_key_iv: "00c01259669d543da35698bfaea0286b"
server_key_version: "encryptionkey@0"
I20260929 19:04:47.107612 21262 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 19:04:47.108266 21277 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:04:47.108435 21262 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 19:04:47.108530 21262 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790708447806542-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790708447806542-392-0/minicluster-data/master-0/wal
uuid: "0afc723a71b34f758453772cd66fd5d2"
format_stamp: "Formatted at 2026-09-29 19:04:47 on dist-test-slave-lqkv"
server_key: "ff0bc3b3b15ef0353a88b96173abb092"
server_key_iv: "00c01259669d543da35698bfaea0286b"
server_key_version: "encryptionkey@0"
I20260929 19:04:47.108608 21262 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790708447806542-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790708447806542-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790708447806542-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:04:47.117478 21262 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:04:47.119083 21262 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:04:47.119243 21262 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:04:47.122987 21262 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45893
I20260929 19:04:47.123031 21329 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45893 every 8 connection(s)
I20260929 19:04:47.123322 21262 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790708447806542-392-0/minicluster-data/master-0/data/info.pb
I20260929 19:04:47.124071 21330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:04:47.125973 21330 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0afc723a71b34f758453772cd66fd5d2: Bootstrap starting.
I20260929 19:04:47.126498 21330 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0afc723a71b34f758453772cd66fd5d2: Neither blocks nor log segments found. Creating new log.
I20260929 19:04:47.126793 21330 log.cc:824] T 00000000000000000000000000000000 P 0afc723a71b34f758453772cd66fd5d2: Log is configured to *not* fsync() on all Append() calls
I20260929 19:04:47.127416 21330 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0afc723a71b34f758453772cd66fd5d2: No bootstrap required, opened a new log
I20260929 19:04:47.128959 21330 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0afc723a71b34f758453772cd66fd5d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0afc723a71b34f758453772cd66fd5d2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45893 } }
I20260929 19:04:47.129068 21330 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0afc723a71b34f758453772cd66fd5d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:04:47.129088 21330 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0afc723a71b34f758453772cd66fd5d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0afc723a71b34f758453772cd66fd5d2, State: Initialized, Role: FOLLOWER
I20260929 19:04:47.129230 21330 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0afc723a71b34f758453772cd66fd5d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0afc723a71b34f758453772cd66fd5d2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45893 } }
I20260929 19:04:47.129307 21330 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0afc723a71b34f758453772cd66fd5d2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 19:04:47.129345 21330 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0afc723a71b34f758453772cd66fd5d2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 19:04:47.129390 21330 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0afc723a71b34f758453772cd66fd5d2 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:04:47.129853 21330 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0afc723a71b34f758453772cd66fd5d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0afc723a71b34f758453772cd66fd5d2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45893 } }
I20260929 19:04:47.129930 21330 leader_election.cc:302] T 00000000000000000000000000000000 P 0afc723a71b34f758453772cd66fd5d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0afc723a71b34f758453772cd66fd5d2; no voters:
I20260929 19:04:47.130118 21330 leader_election.cc:288] T 00000000000000000000000000000000 P 0afc723a71b34f758453772cd66fd5d2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 19:04:47.130158 21333 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0afc723a71b34f758453772cd66fd5d2 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:04:47.130319 21333 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0afc723a71b34f758453772cd66fd5d2 [term 1 LEADER]: Becoming Leader. State: Replica: 0afc723a71b34f758453772cd66fd5d2, State: Running, Role: LEADER
I20260929 19:04:47.130347 21330 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0afc723a71b34f758453772cd66fd5d2 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 19:04:47.130400 21333 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0afc723a71b34f758453772cd66fd5d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0afc723a71b34f758453772cd66fd5d2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45893 } }
I20260929 19:04:47.130784 21334 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0afc723a71b34f758453772cd66fd5d2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0afc723a71b34f758453772cd66fd5d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0afc723a71b34f758453772cd66fd5d2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45893 } } }
I20260929 19:04:47.130795 21335 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0afc723a71b34f758453772cd66fd5d2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0afc723a71b34f758453772cd66fd5d2. Latest consensus state: current_term: 1 leader_uuid: "0afc723a71b34f758453772cd66fd5d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0afc723a71b34f758453772cd66fd5d2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45893 } } }
I20260929 19:04:47.130920 21335 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0afc723a71b34f758453772cd66fd5d2 [sys.catalog]: This master's current role is: LEADER
I20260929 19:04:47.130883 21334 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0afc723a71b34f758453772cd66fd5d2 [sys.catalog]: This master's current role is: LEADER
I20260929 19:04:47.131121 21338 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 19:04:47.131676 21338 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 19:04:47.132984 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUbnLHE/build/release/bin/kudu as pid 21262
I20260929 19:04:47.133098 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790708447806542-392-0/minicluster-data/master-0/wal/instance
I20260929 19:04:47.133131 21338 catalog_manager.cc:1347] Generated new cluster ID: 5e72f9e1b27942ed826e29048e1460e8
I20260929 19:04:47.133168 21338 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 19:04:47.133239 392 external_mini_cluster.cc:1428] Setting key d521e9999b74da1f10a2934b59819ab8
Sep 29 19:04:47 dist-test-slave-lqkv krb5kdc[21249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708686, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 19:04:47.142341 21338 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 19:04:47.142984 21338 catalog_manager.cc:1504] Loading token signing keys...
I20260929 19:04:47.143950 21353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:47.133631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.111:36187 (local address 127.0.0.1:45893)
0929 19:04:47.133779 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0929 19:04:47.133787 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 19:04:47.133836 (+ 49us) server_negotiation.cc:407] Connection header received
0929 19:04:47.133926 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:04:47.133932 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:04:47.133977 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:04:47.134079 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:04:47.134835 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:47.136193 (+ 1358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:47.137277 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:47.137863 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:47.137957 (+ 94us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:04:47.140447 (+ 2490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:04:47.140450 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:04:47.140451 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:04:47.140461 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:04:47.142336 (+ 1875us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:04:47.142780 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:04:47.142783 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:04:47.142783 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:04:47.142879 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:04:47.143157 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:04:47.143158 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:04:47.143158 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:04:47.143384 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:04:47.143533 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0929 19:04:47.143660 (+ 127us) server_negotiation.cc:299] Negotiation successful
0929 19:04:47.143751 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":63,"threads_started":1}
I20260929 19:04:47.144773 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 19:04:47.151227 21338 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0afc723a71b34f758453772cd66fd5d2: Generated new TSK 0
I20260929 19:04:47.151378 21338 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 19:04:47.157466 21353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:47.146128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.111:58801 (local address 127.0.0.1:45893)
0929 19:04:47.146159 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0929 19:04:47.146161 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:04:47.151013 (+ 4852us) server_negotiation.cc:407] Connection header received
0929 19:04:47.151124 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:04:47.151125 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:04:47.151133 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:04:47.151193 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:04:47.151953 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:47.153002 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:47.153642 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:47.153968 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:47.154042 (+ 74us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:04:47.155106 (+ 1064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:04:47.155107 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:04:47.155107 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:04:47.155113 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:04:47.156414 (+ 1301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:04:47.156744 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:04:47.156745 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:04:47.156745 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:04:47.156799 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:04:47.157016 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:04:47.157017 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:04:47.157018 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:04:47.157129 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:04:47.157224 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0929 19:04:47.157312 (+ 88us) server_negotiation.cc:299] Negotiation successful
0929 19:04:47.157368 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260929 19:04:47.159179 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUbnLHE/build/release/bin/kudu with pid 21262
W20260929 19:04:47.163019 21247 connection.cc:535] client connection to 127.0.0.1:45893 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45893): Connection reset by peer (error 104)
2026-09-29T19:04:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1205 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790708447806542-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 19:04:47 dist-test-slave-lqkv krb5kdc[21361](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 19:04:47 dist-test-slave-lqkv krb5kdc[21361](info): set up 2 sockets
Sep 29 19:04:47 dist-test-slave-lqkv krb5kdc[21361](info): commencing operation
krb5kdc: starting...
W20260929 19:04:48.128575 392 mini_kdc.cc:121] Time spent starting KDC: real 0.952s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790708447806542-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790708447806542-392-0/krb5kdc/test-user.keytab.
Sep 29 19:04:48 dist-test-slave-lqkv krb5kdc[21361](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708688, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T19:04:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T19:04:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790708447806542-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790708447806542-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 19:04:48.232995 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUbnLHE/build/release/bin/kudu
/tmp/dist-test-taskUbnLHE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790708447806542-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790708447806542-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790708447806542-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790708447806542-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43703
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790708447806542-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43703
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790708447806542-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790708447806542-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790708447806542-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:04:48.305364 21374 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:04:48.305543 21374 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:04:48.305562 21374 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:04:48.306551 21374 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 19:04:48.306599 21374 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:04:48.306612 21374 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:04:48.306653 21374 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 19:04:48.306674 21374 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 19:04:48.308025 21374 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790708447806542-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790708447806542-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43703
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790708447806542-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43703
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790708447806542-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21374
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790708447806542-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 18:09:13 UTC on ccycloud.quasar-hpkalf.root.comops.site
build id 4680913
I20260929 19:04:48.308254 21374 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:04:48.308466 21374 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:04:48.312920 21379 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:04:48.312917 21380 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:04:48.312920 21382 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:04:48.312966 21374 server_base.cc:1034] running on GCE node
I20260929 19:04:48.313323 21374 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:04:48.313647 21374 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:04:48.314806 21374 hybrid_clock.cc:648] HybridClock initialized: now 1790708688314794 us; error 40 us; skew 500 ppm
Sep 29 19:04:48 dist-test-slave-lqkv krb5kdc[21361](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708688, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:04:48.317052 21374 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 19:04:48.317462 21374 webserver.cc:460] Webserver started at http://127.0.0.1:46799/ using document root <none> and password file <none>
I20260929 19:04:48.317682 21374 fs_manager.cc:359] Metadata directory not provided
I20260929 19:04:48.317747 21374 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:04:48.317891 21374 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:04:48.318851 21374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790708447806542-392-0/minicluster-data/master-0/data/instance:
uuid: "5478adc7059a4c338959112bd83faeab"
format_stamp: "Formatted at 2026-09-29 19:04:48 on dist-test-slave-lqkv"
server_key: "3e1e75a1659c4d8744e9c1c0fc2a2264"
server_key_iv: "a0b47c33165bd00b2efe16cfd6100a61"
server_key_version: "encryptionkey@0"
I20260929 19:04:48.319221 21374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790708447806542-392-0/minicluster-data/master-0/wal/instance:
uuid: "5478adc7059a4c338959112bd83faeab"
format_stamp: "Formatted at 2026-09-29 19:04:48 on dist-test-slave-lqkv"
server_key: "3e1e75a1659c4d8744e9c1c0fc2a2264"
server_key_iv: "a0b47c33165bd00b2efe16cfd6100a61"
server_key_version: "encryptionkey@0"
I20260929 19:04:48.320394 21374 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 19:04:48.321046 21389 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:04:48.321209 21374 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 19:04:48.321293 21374 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790708447806542-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790708447806542-392-0/minicluster-data/master-0/wal
uuid: "5478adc7059a4c338959112bd83faeab"
format_stamp: "Formatted at 2026-09-29 19:04:48 on dist-test-slave-lqkv"
server_key: "3e1e75a1659c4d8744e9c1c0fc2a2264"
server_key_iv: "a0b47c33165bd00b2efe16cfd6100a61"
server_key_version: "encryptionkey@0"
I20260929 19:04:48.321362 21374 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790708447806542-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790708447806542-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790708447806542-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:04:48.333983 21374 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:04:48.335489 21374 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:04:48.335635 21374 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:04:48.339228 21374 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43703
I20260929 19:04:48.339257 21441 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43703 every 8 connection(s)
I20260929 19:04:48.339531 21374 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790708447806542-392-0/minicluster-data/master-0/data/info.pb
I20260929 19:04:48.340186 21442 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:04:48.341965 21442 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5478adc7059a4c338959112bd83faeab: Bootstrap starting.
I20260929 19:04:48.342397 21442 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5478adc7059a4c338959112bd83faeab: Neither blocks nor log segments found. Creating new log.
I20260929 19:04:48.342607 21442 log.cc:824] T 00000000000000000000000000000000 P 5478adc7059a4c338959112bd83faeab: Log is configured to *not* fsync() on all Append() calls
I20260929 19:04:48.343132 21442 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5478adc7059a4c338959112bd83faeab: No bootstrap required, opened a new log
I20260929 19:04:48.344476 21442 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5478adc7059a4c338959112bd83faeab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5478adc7059a4c338959112bd83faeab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43703 } }
I20260929 19:04:48.344584 21442 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5478adc7059a4c338959112bd83faeab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:04:48.344604 21442 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5478adc7059a4c338959112bd83faeab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5478adc7059a4c338959112bd83faeab, State: Initialized, Role: FOLLOWER
I20260929 19:04:48.344714 21442 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5478adc7059a4c338959112bd83faeab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5478adc7059a4c338959112bd83faeab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43703 } }
I20260929 19:04:48.344782 21442 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5478adc7059a4c338959112bd83faeab [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 19:04:48.344853 21442 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5478adc7059a4c338959112bd83faeab [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 19:04:48.344895 21442 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5478adc7059a4c338959112bd83faeab [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:04:48.345299 21442 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5478adc7059a4c338959112bd83faeab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5478adc7059a4c338959112bd83faeab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43703 } }
I20260929 19:04:48.345367 21442 leader_election.cc:302] T 00000000000000000000000000000000 P 5478adc7059a4c338959112bd83faeab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5478adc7059a4c338959112bd83faeab; no voters:
I20260929 19:04:48.345475 21442 leader_election.cc:288] T 00000000000000000000000000000000 P 5478adc7059a4c338959112bd83faeab [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 19:04:48.345547 21445 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5478adc7059a4c338959112bd83faeab [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:04:48.345677 21445 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5478adc7059a4c338959112bd83faeab [term 1 LEADER]: Becoming Leader. State: Replica: 5478adc7059a4c338959112bd83faeab, State: Running, Role: LEADER
I20260929 19:04:48.345692 21442 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5478adc7059a4c338959112bd83faeab [sys.catalog]: configured and running, proceeding with master startup.
I20260929 19:04:48.345753 21445 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5478adc7059a4c338959112bd83faeab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5478adc7059a4c338959112bd83faeab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43703 } }
I20260929 19:04:48.346091 21446 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5478adc7059a4c338959112bd83faeab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5478adc7059a4c338959112bd83faeab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5478adc7059a4c338959112bd83faeab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43703 } } }
I20260929 19:04:48.346119 21447 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5478adc7059a4c338959112bd83faeab [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5478adc7059a4c338959112bd83faeab. Latest consensus state: current_term: 1 leader_uuid: "5478adc7059a4c338959112bd83faeab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5478adc7059a4c338959112bd83faeab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43703 } } }
I20260929 19:04:48.346436 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUbnLHE/build/release/bin/kudu as pid 21374
I20260929 19:04:48.346557 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790708447806542-392-0/minicluster-data/master-0/wal/instance
I20260929 19:04:48.346743 392 external_mini_cluster.cc:1428] Setting key 14345f8b4fb667ad6ec3ebead600084e
I20260929 19:04:48.346226 21446 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5478adc7059a4c338959112bd83faeab [sys.catalog]: This master's current role is: LEADER
I20260929 19:04:48.346247 21447 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5478adc7059a4c338959112bd83faeab [sys.catalog]: This master's current role is: LEADER
I20260929 19:04:48.347259 21460 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 19:04:48.347888 21460 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 19:04:48.349268 21460 catalog_manager.cc:1347] Generated new cluster ID: 254d9738f43843639feb00334e8f4ca1
I20260929 19:04:48.349316 21460 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 19:04:48.352730 21463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:48.347298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.111:43625 (local address 127.0.0.1:43703)
0929 19:04:48.347479 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0929 19:04:48.347493 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 19:04:48.347540 (+ 47us) server_negotiation.cc:407] Connection header received
0929 19:04:48.347703 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:04:48.347711 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:04:48.347777 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:04:48.347923 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:04:48.348197 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:48.349539 (+ 1342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:48.350663 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:48.351215 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:48.351305 (+ 90us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:04:48.352035 (+ 730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:04:48.352038 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:04:48.352039 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 19:04:48.352217 (+ 178us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:04:48.352249 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 19:04:48.352262 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:04:48.352337 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 19:04:48.352518 (+ 181us) server_negotiation.cc:299] Negotiation successful
0929 19:04:48.352542 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":117,"threads_started":1}
I20260929 19:04:48.353580 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 19:04:48.358331 21463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:48.354902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.111:39305 (local address 127.0.0.1:43703)
0929 19:04:48.354933 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0929 19:04:48.354935 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:04:48.355069 (+ 134us) server_negotiation.cc:407] Connection header received
0929 19:04:48.355166 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:04:48.355167 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:04:48.355177 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:04:48.355208 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:04:48.355399 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:48.356064 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:48.357037 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:48.357344 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:48.357381 (+ 37us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:04:48.358077 (+ 696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:04:48.358078 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:04:48.358078 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 19:04:48.358083 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:04:48.358095 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 19:04:48.358099 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:04:48.358131 (+ 32us) server_negotiation.cc:1035] Waiting for connection context
0929 19:04:48.358257 (+ 126us) server_negotiation.cc:299] Negotiation successful
0929 19:04:48.358268 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260929 19:04:48.363382 21460 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 19:04:48.363839 21460 catalog_manager.cc:1504] Loading token signing keys...
I20260929 19:04:48.368165 21460 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5478adc7059a4c338959112bd83faeab: Generated new TSK 0
I20260929 19:04:48.368332 21460 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 19:04:48.372939 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUbnLHE/build/release/bin/kudu with pid 21374
W20260929 19:04:48.376717 21359 connection.cc:535] client connection to 127.0.0.1:43703 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43703): Connection reset by peer (error 104)
2026-09-29T19:04:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1213 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790708447806542-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 19:04:48 dist-test-slave-lqkv krb5kdc[21473](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 19:04:48 dist-test-slave-lqkv krb5kdc[21473](info): set up 2 sockets
Sep 29 19:04:48 dist-test-slave-lqkv krb5kdc[21473](info): commencing operation
krb5kdc: starting...
W20260929 19:04:49.348704 392 mini_kdc.cc:121] Time spent starting KDC: real 0.958s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790708447806542-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790708447806542-392-0/krb5kdc/test-user.keytab.
Sep 29 19:04:49 dist-test-slave-lqkv krb5kdc[21473](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708689, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T19:04:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T19:04:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790708447806542-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790708447806542-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 19:04:49.450945 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUbnLHE/build/release/bin/kudu
/tmp/dist-test-taskUbnLHE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790708447806542-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790708447806542-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790708447806542-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790708447806542-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36111
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790708447806542-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36111
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790708447806542-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790708447806542-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790708447806542-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:04:49.523983 21486 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:04:49.524196 21486 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:04:49.524230 21486 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:04:49.525296 21486 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 19:04:49.525373 21486 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:04:49.525395 21486 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:04:49.525413 21486 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 19:04:49.525439 21486 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 19:04:49.526695 21486 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790708447806542-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790708447806542-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36111
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790708447806542-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36111
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790708447806542-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21486
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790708447806542-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 18:09:13 UTC on ccycloud.quasar-hpkalf.root.comops.site
build id 4680913
I20260929 19:04:49.526939 21486 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:04:49.527129 21486 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:04:49.531383 21494 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:04:49.531414 21492 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:04:49.531418 21491 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:04:49.531597 21486 server_base.cc:1034] running on GCE node
I20260929 19:04:49.531797 21486 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:04:49.532034 21486 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:04:49.533161 21486 hybrid_clock.cc:648] HybridClock initialized: now 1790708689533148 us; error 27 us; skew 500 ppm
Sep 29 19:04:49 dist-test-slave-lqkv krb5kdc[21473](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708689, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:04:49.535231 21486 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 19:04:49.535630 21486 webserver.cc:460] Webserver started at http://127.0.0.1:39349/ using document root <none> and password file <none>
I20260929 19:04:49.535847 21486 fs_manager.cc:359] Metadata directory not provided
I20260929 19:04:49.535888 21486 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:04:49.536021 21486 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:04:49.537014 21486 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790708447806542-392-0/minicluster-data/master-0/data/instance:
uuid: "5c04fe4cbfee45719d4eb31b12ef7531"
format_stamp: "Formatted at 2026-09-29 19:04:49 on dist-test-slave-lqkv"
server_key: "3a6c05cff635e64f6da954718c83c247"
server_key_iv: "83deff10825fce14ce85e9ecfa9da7f0"
server_key_version: "encryptionkey@0"
I20260929 19:04:49.537334 21486 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790708447806542-392-0/minicluster-data/master-0/wal/instance:
uuid: "5c04fe4cbfee45719d4eb31b12ef7531"
format_stamp: "Formatted at 2026-09-29 19:04:49 on dist-test-slave-lqkv"
server_key: "3a6c05cff635e64f6da954718c83c247"
server_key_iv: "83deff10825fce14ce85e9ecfa9da7f0"
server_key_version: "encryptionkey@0"
I20260929 19:04:49.538555 21486 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260929 19:04:49.539230 21501 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:04:49.539376 21486 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 19:04:49.539469 21486 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790708447806542-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790708447806542-392-0/minicluster-data/master-0/wal
uuid: "5c04fe4cbfee45719d4eb31b12ef7531"
format_stamp: "Formatted at 2026-09-29 19:04:49 on dist-test-slave-lqkv"
server_key: "3a6c05cff635e64f6da954718c83c247"
server_key_iv: "83deff10825fce14ce85e9ecfa9da7f0"
server_key_version: "encryptionkey@0"
I20260929 19:04:49.539541 21486 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790708447806542-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790708447806542-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790708447806542-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:04:49.542534 21486 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:04:49.544188 21486 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:04:49.544332 21486 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:04:49.547888 21486 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36111
I20260929 19:04:49.547916 21553 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36111 every 8 connection(s)
I20260929 19:04:49.548199 21486 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790708447806542-392-0/minicluster-data/master-0/data/info.pb
I20260929 19:04:49.549175 21554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:04:49.551111 21554 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c04fe4cbfee45719d4eb31b12ef7531: Bootstrap starting.
I20260929 19:04:49.551610 21554 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5c04fe4cbfee45719d4eb31b12ef7531: Neither blocks nor log segments found. Creating new log.
I20260929 19:04:49.551873 21554 log.cc:824] T 00000000000000000000000000000000 P 5c04fe4cbfee45719d4eb31b12ef7531: Log is configured to *not* fsync() on all Append() calls
I20260929 19:04:49.552467 21554 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c04fe4cbfee45719d4eb31b12ef7531: No bootstrap required, opened a new log
I20260929 19:04:49.553946 21554 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5c04fe4cbfee45719d4eb31b12ef7531 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c04fe4cbfee45719d4eb31b12ef7531" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36111 } }
I20260929 19:04:49.554061 21554 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5c04fe4cbfee45719d4eb31b12ef7531 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:04:49.554082 21554 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5c04fe4cbfee45719d4eb31b12ef7531 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c04fe4cbfee45719d4eb31b12ef7531, State: Initialized, Role: FOLLOWER
I20260929 19:04:49.554204 21554 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5c04fe4cbfee45719d4eb31b12ef7531 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c04fe4cbfee45719d4eb31b12ef7531" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36111 } }
I20260929 19:04:49.554278 21554 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5c04fe4cbfee45719d4eb31b12ef7531 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 19:04:49.554301 21554 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5c04fe4cbfee45719d4eb31b12ef7531 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 19:04:49.554332 21554 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5c04fe4cbfee45719d4eb31b12ef7531 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:04:49.554638 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUbnLHE/build/release/bin/kudu as pid 21486
I20260929 19:04:49.554725 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790708447806542-392-0/minicluster-data/master-0/wal/instance
I20260929 19:04:49.554853 21554 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5c04fe4cbfee45719d4eb31b12ef7531 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c04fe4cbfee45719d4eb31b12ef7531" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36111 } }
I20260929 19:04:49.554957 392 external_mini_cluster.cc:1428] Setting key 10462fe5dc1fcc6547837e5ba6a9e86d
I20260929 19:04:49.554963 21554 leader_election.cc:302] T 00000000000000000000000000000000 P 5c04fe4cbfee45719d4eb31b12ef7531 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5c04fe4cbfee45719d4eb31b12ef7531; no voters:
I20260929 19:04:49.555146 21554 leader_election.cc:288] T 00000000000000000000000000000000 P 5c04fe4cbfee45719d4eb31b12ef7531 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 19:04:49.555197 21557 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5c04fe4cbfee45719d4eb31b12ef7531 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:04:49.555342 21557 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5c04fe4cbfee45719d4eb31b12ef7531 [term 1 LEADER]: Becoming Leader. State: Replica: 5c04fe4cbfee45719d4eb31b12ef7531, State: Running, Role: LEADER
I20260929 19:04:49.555382 21554 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5c04fe4cbfee45719d4eb31b12ef7531 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 19:04:49.555418 21557 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5c04fe4cbfee45719d4eb31b12ef7531 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c04fe4cbfee45719d4eb31b12ef7531" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36111 } }
I20260929 19:04:49.555703 21557 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c04fe4cbfee45719d4eb31b12ef7531 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5c04fe4cbfee45719d4eb31b12ef7531. Latest consensus state: current_term: 1 leader_uuid: "5c04fe4cbfee45719d4eb31b12ef7531" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c04fe4cbfee45719d4eb31b12ef7531" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36111 } } }
I20260929 19:04:49.555719 21559 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c04fe4cbfee45719d4eb31b12ef7531 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5c04fe4cbfee45719d4eb31b12ef7531" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c04fe4cbfee45719d4eb31b12ef7531" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36111 } } }
I20260929 19:04:49.555816 21557 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c04fe4cbfee45719d4eb31b12ef7531 [sys.catalog]: This master's current role is: LEADER
I20260929 19:04:49.555876 21559 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c04fe4cbfee45719d4eb31b12ef7531 [sys.catalog]: This master's current role is: LEADER
I20260929 19:04:49.556070 21563 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 19:04:49.556594 21563 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 19:04:49.558288 21563 catalog_manager.cc:1347] Generated new cluster ID: 4e1a4c5ca89b4cacb9e626d3afa6ba1b
I20260929 19:04:49.558363 21563 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 19:04:49.558527 21561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:49.555364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.111:37475 (local address 127.0.0.1:36111)
0929 19:04:49.555523 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0929 19:04:49.555534 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 19:04:49.557121 (+ 1587us) server_negotiation.cc:407] Connection header received
0929 19:04:49.557268 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:04:49.557273 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:04:49.557294 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:04:49.557416 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:04:49.558035 (+ 619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:04:49.558039 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:04:49.558039 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 19:04:49.558043 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:04:49.558079 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 19:04:49.558097 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:04:49.558169 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 19:04:49.558241 (+ 72us) server_negotiation.cc:299] Negotiation successful
0929 19:04:49.558285 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":77,"threads_started":1}
I20260929 19:04:49.559312 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 19:04:49.561924 21561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:49.560573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.111:35839 (local address 127.0.0.1:36111)
0929 19:04:49.560608 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0929 19:04:49.560610 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:04:49.560964 (+ 354us) server_negotiation.cc:407] Connection header received
0929 19:04:49.561036 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:04:49.561036 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:04:49.561041 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:04:49.561089 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:04:49.561688 (+ 599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:04:49.561689 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:04:49.561690 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 19:04:49.561694 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:04:49.561706 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 19:04:49.561709 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:04:49.561763 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0929 19:04:49.561815 (+ 52us) server_negotiation.cc:299] Negotiation successful
0929 19:04:49.561855 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260929 19:04:49.566570 21563 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 19:04:49.566704 21563 catalog_manager.cc:1504] Loading token signing keys...
I20260929 19:04:49.569756 21563 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5c04fe4cbfee45719d4eb31b12ef7531: Generated new TSK 0
I20260929 19:04:49.569911 21563 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 19:04:49.575587 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUbnLHE/build/release/bin/kudu with pid 21486
2026-09-29T19:04:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1202 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790708447806542-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 19:04:49 dist-test-slave-lqkv krb5kdc[21585](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 19:04:49 dist-test-slave-lqkv krb5kdc[21585](info): set up 2 sockets
Sep 29 19:04:49 dist-test-slave-lqkv krb5kdc[21585](info): commencing operation
krb5kdc: starting...
W20260929 19:04:50.532418 392 mini_kdc.cc:121] Time spent starting KDC: real 0.939s user 0.001s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790708447806542-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790708447806542-392-0/krb5kdc/test-user.keytab.
Sep 29 19:04:50 dist-test-slave-lqkv krb5kdc[21585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708690, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T19:04:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T19:04:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790708447806542-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790708447806542-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 19:04:50.633571 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUbnLHE/build/release/bin/kudu
/tmp/dist-test-taskUbnLHE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790708447806542-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790708447806542-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790708447806542-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790708447806542-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35125
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790708447806542-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35125
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790708447806542-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790708447806542-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790708447806542-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:04:50.709106 21598 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:04:50.709302 21598 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:04:50.709367 21598 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:04:50.710371 21598 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 19:04:50.710430 21598 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:04:50.710453 21598 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:04:50.710479 21598 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 19:04:50.710500 21598 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 19:04:50.711887 21598 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790708447806542-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790708447806542-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35125
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790708447806542-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35125
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790708447806542-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21598
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790708447806542-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 18:09:13 UTC on ccycloud.quasar-hpkalf.root.comops.site
build id 4680913
I20260929 19:04:50.712152 21598 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:04:50.712373 21598 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:04:50.716643 21604 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:04:50.716701 21603 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:04:50.716838 21598 server_base.cc:1034] running on GCE node
W20260929 19:04:50.716887 21606 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:04:50.717072 21598 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:04:50.717311 21598 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:04:50.718472 21598 hybrid_clock.cc:648] HybridClock initialized: now 1790708690718464 us; error 36 us; skew 500 ppm
Sep 29 19:04:50 dist-test-slave-lqkv krb5kdc[21585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708690, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:04:50.720695 21598 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 19:04:50.721138 21598 webserver.cc:460] Webserver started at http://127.0.0.1:42855/ using document root <none> and password file <none>
I20260929 19:04:50.721364 21598 fs_manager.cc:359] Metadata directory not provided
I20260929 19:04:50.721432 21598 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:04:50.721575 21598 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:04:50.722507 21598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790708447806542-392-0/minicluster-data/master-0/data/instance:
uuid: "7eefe3de38fa48bbbb5f0f556876388b"
format_stamp: "Formatted at 2026-09-29 19:04:50 on dist-test-slave-lqkv"
server_key: "93af65ce29723212c56a36591628319d"
server_key_iv: "134ec9088e6ecfba8236df8aa12db9b5"
server_key_version: "encryptionkey@0"
I20260929 19:04:50.722831 21598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790708447806542-392-0/minicluster-data/master-0/wal/instance:
uuid: "7eefe3de38fa48bbbb5f0f556876388b"
format_stamp: "Formatted at 2026-09-29 19:04:50 on dist-test-slave-lqkv"
server_key: "93af65ce29723212c56a36591628319d"
server_key_iv: "134ec9088e6ecfba8236df8aa12db9b5"
server_key_version: "encryptionkey@0"
I20260929 19:04:50.724007 21598 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 19:04:50.724649 21613 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:04:50.724834 21598 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 19:04:50.724910 21598 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790708447806542-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790708447806542-392-0/minicluster-data/master-0/wal
uuid: "7eefe3de38fa48bbbb5f0f556876388b"
format_stamp: "Formatted at 2026-09-29 19:04:50 on dist-test-slave-lqkv"
server_key: "93af65ce29723212c56a36591628319d"
server_key_iv: "134ec9088e6ecfba8236df8aa12db9b5"
server_key_version: "encryptionkey@0"
I20260929 19:04:50.724987 21598 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790708447806542-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790708447806542-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790708447806542-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:04:50.733336 21598 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:04:50.735284 21598 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:04:50.735455 21598 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:04:50.739151 21598 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35125
I20260929 19:04:50.739195 21665 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35125 every 8 connection(s)
I20260929 19:04:50.739502 21598 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790708447806542-392-0/minicluster-data/master-0/data/info.pb
I20260929 19:04:50.740207 21666 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:04:50.742264 21666 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7eefe3de38fa48bbbb5f0f556876388b: Bootstrap starting.
I20260929 19:04:50.742729 21666 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7eefe3de38fa48bbbb5f0f556876388b: Neither blocks nor log segments found. Creating new log.
I20260929 19:04:50.742935 21666 log.cc:824] T 00000000000000000000000000000000 P 7eefe3de38fa48bbbb5f0f556876388b: Log is configured to *not* fsync() on all Append() calls
I20260929 19:04:50.743480 21666 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7eefe3de38fa48bbbb5f0f556876388b: No bootstrap required, opened a new log
I20260929 19:04:50.744777 21666 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7eefe3de38fa48bbbb5f0f556876388b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7eefe3de38fa48bbbb5f0f556876388b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35125 } }
I20260929 19:04:50.744952 21666 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7eefe3de38fa48bbbb5f0f556876388b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:04:50.744976 21666 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7eefe3de38fa48bbbb5f0f556876388b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7eefe3de38fa48bbbb5f0f556876388b, State: Initialized, Role: FOLLOWER
I20260929 19:04:50.745085 21666 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7eefe3de38fa48bbbb5f0f556876388b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7eefe3de38fa48bbbb5f0f556876388b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35125 } }
I20260929 19:04:50.745157 21666 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7eefe3de38fa48bbbb5f0f556876388b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 19:04:50.745177 21666 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7eefe3de38fa48bbbb5f0f556876388b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 19:04:50.745206 21666 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7eefe3de38fa48bbbb5f0f556876388b [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:04:50.745611 21666 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7eefe3de38fa48bbbb5f0f556876388b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7eefe3de38fa48bbbb5f0f556876388b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35125 } }
I20260929 19:04:50.745682 21666 leader_election.cc:302] T 00000000000000000000000000000000 P 7eefe3de38fa48bbbb5f0f556876388b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7eefe3de38fa48bbbb5f0f556876388b; no voters:
I20260929 19:04:50.745780 21666 leader_election.cc:288] T 00000000000000000000000000000000 P 7eefe3de38fa48bbbb5f0f556876388b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 19:04:50.745895 21669 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7eefe3de38fa48bbbb5f0f556876388b [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:04:50.746011 21666 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7eefe3de38fa48bbbb5f0f556876388b [sys.catalog]: configured and running, proceeding with master startup.
I20260929 19:04:50.746043 21669 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7eefe3de38fa48bbbb5f0f556876388b [term 1 LEADER]: Becoming Leader. State: Replica: 7eefe3de38fa48bbbb5f0f556876388b, State: Running, Role: LEADER
I20260929 19:04:50.746141 21669 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7eefe3de38fa48bbbb5f0f556876388b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7eefe3de38fa48bbbb5f0f556876388b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35125 } }
I20260929 19:04:50.746481 21671 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7eefe3de38fa48bbbb5f0f556876388b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7eefe3de38fa48bbbb5f0f556876388b. Latest consensus state: current_term: 1 leader_uuid: "7eefe3de38fa48bbbb5f0f556876388b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7eefe3de38fa48bbbb5f0f556876388b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35125 } } }
I20260929 19:04:50.746521 21670 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7eefe3de38fa48bbbb5f0f556876388b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7eefe3de38fa48bbbb5f0f556876388b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7eefe3de38fa48bbbb5f0f556876388b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35125 } } }
I20260929 19:04:50.746613 21671 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7eefe3de38fa48bbbb5f0f556876388b [sys.catalog]: This master's current role is: LEADER
I20260929 19:04:50.746626 21670 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7eefe3de38fa48bbbb5f0f556876388b [sys.catalog]: This master's current role is: LEADER
I20260929 19:04:50.747159 21675 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 19:04:50.747558 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUbnLHE/build/release/bin/kudu as pid 21598
I20260929 19:04:50.747671 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790708447806542-392-0/minicluster-data/master-0/wal/instance
I20260929 19:04:50.747807 21675 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 19:04:50.747855 392 external_mini_cluster.cc:1428] Setting key b9854fe403581838ef401c733c021bb7
I20260929 19:04:50.749598 21675 catalog_manager.cc:1347] Generated new cluster ID: 58cb8702f4014b13b779a074caacd705
I20260929 19:04:50.749645 21675 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 29 19:04:50 dist-test-slave-lqkv krb5kdc[21585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708690, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 19:04:50.758536 21688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:50.748522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.111:40749 (local address 127.0.0.1:35125)
0929 19:04:50.748677 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0929 19:04:50.748686 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 19:04:50.748714 (+ 28us) server_negotiation.cc:407] Connection header received
0929 19:04:50.748842 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:04:50.748848 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:04:50.748904 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:04:50.749789 (+ 885us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:04:50.749868 (+ 79us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:50.751167 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:50.752154 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:50.752747 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:50.752867 (+ 120us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:04:50.755192 (+ 2325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:04:50.755196 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:04:50.755196 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:04:50.755206 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:04:50.756970 (+ 1764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:04:50.757344 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:04:50.757353 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:04:50.757353 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:04:50.757457 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:04:50.757665 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:04:50.757666 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:04:50.757666 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:04:50.757968 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:04:50.758118 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0929 19:04:50.758220 (+ 102us) server_negotiation.cc:299] Negotiation successful
0929 19:04:50.758312 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":78,"threads_started":1}
I20260929 19:04:50.759310 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 19:04:50.767129 21688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:50.760568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.111:53215 (local address 127.0.0.1:35125)
0929 19:04:50.760599 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0929 19:04:50.760601 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:04:50.760637 (+ 36us) server_negotiation.cc:407] Connection header received
0929 19:04:50.760686 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:04:50.760687 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:04:50.760695 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:04:50.760755 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:04:50.761344 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:50.762109 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:50.762996 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:50.763308 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:50.763387 (+ 79us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:04:50.764595 (+ 1208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:04:50.764596 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:04:50.764596 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:04:50.764603 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:04:50.765999 (+ 1396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:04:50.766365 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:04:50.766366 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:04:50.766366 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:04:50.766438 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:04:50.766644 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:04:50.766645 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:04:50.766645 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:04:50.766762 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:04:50.766880 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0929 19:04:50.766994 (+ 114us) server_negotiation.cc:299] Negotiation successful
0929 19:04:50.767050 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260929 19:04:50.769167 21675 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 19:04:50.769588 21675 catalog_manager.cc:1504] Loading token signing keys...
I20260929 19:04:50.773115 21675 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7eefe3de38fa48bbbb5f0f556876388b: Generated new TSK 0
I20260929 19:04:50.773291 21675 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 19:04:50.781432 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUbnLHE/build/release/bin/kudu with pid 21598
W20260929 19:04:50.784996 21583 connection.cc:535] client connection to 127.0.0.1:35125 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35125): Connection reset by peer (error 104)
2026-09-29T19:04:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1205 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790708447806542-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 19:04:50 dist-test-slave-lqkv krb5kdc[21697](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 19:04:50 dist-test-slave-lqkv krb5kdc[21697](info): set up 2 sockets
Sep 29 19:04:50 dist-test-slave-lqkv krb5kdc[21697](info): commencing operation
krb5kdc: starting...
W20260929 19:04:51.769027 392 mini_kdc.cc:121] Time spent starting KDC: real 0.971s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790708447806542-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790708447806542-392-0/krb5kdc/test-user.keytab.
Sep 29 19:04:51 dist-test-slave-lqkv krb5kdc[21697](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708691, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T19:04:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T19:04:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790708447806542-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790708447806542-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 19:04:51.880620 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUbnLHE/build/release/bin/kudu
/tmp/dist-test-taskUbnLHE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790708447806542-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790708447806542-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790708447806542-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790708447806542-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41171
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790708447806542-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41171
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790708447806542-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790708447806542-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790708447806542-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:04:51.955394 21710 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:04:51.955605 21710 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:04:51.955641 21710 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:04:51.956707 21710 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 19:04:51.956768 21710 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:04:51.956794 21710 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:04:51.956816 21710 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 19:04:51.956877 21710 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 19:04:51.958284 21710 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790708447806542-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790708447806542-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41171
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790708447806542-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41171
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790708447806542-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21710
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790708447806542-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 18:09:13 UTC on ccycloud.quasar-hpkalf.root.comops.site
build id 4680913
I20260929 19:04:51.958545 21710 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:04:51.958750 21710 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:04:51.963284 21718 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:04:51.963294 21715 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:04:51.963292 21716 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:04:51.963464 21710 server_base.cc:1034] running on GCE node
I20260929 19:04:51.963725 21710 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:04:51.963990 21710 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:04:51.965142 21710 hybrid_clock.cc:648] HybridClock initialized: now 1790708691965122 us; error 29 us; skew 500 ppm
Sep 29 19:04:51 dist-test-slave-lqkv krb5kdc[21697](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708691, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:04:51.972908 21710 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 19:04:51.973323 21710 webserver.cc:460] Webserver started at http://127.0.0.1:39311/ using document root <none> and password file <none>
I20260929 19:04:51.973544 21710 fs_manager.cc:359] Metadata directory not provided
I20260929 19:04:51.973610 21710 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:04:51.973752 21710 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:04:51.974750 21710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790708447806542-392-0/minicluster-data/master-0/data/instance:
uuid: "cdc5863482cf4c88a6e6212fc5e82604"
format_stamp: "Formatted at 2026-09-29 19:04:51 on dist-test-slave-lqkv"
server_key: "59763e59a9808b5a86b2c02b2395fa77"
server_key_iv: "3acf343ac24c394f8f6257c17dd1fbab"
server_key_version: "encryptionkey@0"
I20260929 19:04:51.975087 21710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790708447806542-392-0/minicluster-data/master-0/wal/instance:
uuid: "cdc5863482cf4c88a6e6212fc5e82604"
format_stamp: "Formatted at 2026-09-29 19:04:51 on dist-test-slave-lqkv"
server_key: "59763e59a9808b5a86b2c02b2395fa77"
server_key_iv: "3acf343ac24c394f8f6257c17dd1fbab"
server_key_version: "encryptionkey@0"
I20260929 19:04:51.976291 21710 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 19:04:51.976980 21725 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:04:51.977142 21710 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 19:04:51.977232 21710 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790708447806542-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790708447806542-392-0/minicluster-data/master-0/wal
uuid: "cdc5863482cf4c88a6e6212fc5e82604"
format_stamp: "Formatted at 2026-09-29 19:04:51 on dist-test-slave-lqkv"
server_key: "59763e59a9808b5a86b2c02b2395fa77"
server_key_iv: "3acf343ac24c394f8f6257c17dd1fbab"
server_key_version: "encryptionkey@0"
I20260929 19:04:51.977312 21710 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790708447806542-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790708447806542-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790708447806542-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:04:51.985877 21710 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:04:51.987473 21710 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:04:51.987618 21710 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:04:51.991523 21710 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41171
I20260929 19:04:51.991552 21777 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41171 every 8 connection(s)
I20260929 19:04:51.991858 21710 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790708447806542-392-0/minicluster-data/master-0/data/info.pb
I20260929 19:04:51.992691 21778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:04:51.994544 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUbnLHE/build/release/bin/kudu as pid 21710
I20260929 19:04:51.994647 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790708447806542-392-0/minicluster-data/master-0/wal/instance
I20260929 19:04:51.994663 21778 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cdc5863482cf4c88a6e6212fc5e82604: Bootstrap starting.
I20260929 19:04:51.994815 392 external_mini_cluster.cc:1428] Setting key 735c147383aaa170ac98ea0109bfd05d
I20260929 19:04:51.995222 21778 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cdc5863482cf4c88a6e6212fc5e82604: Neither blocks nor log segments found. Creating new log.
I20260929 19:04:51.995537 21778 log.cc:824] T 00000000000000000000000000000000 P cdc5863482cf4c88a6e6212fc5e82604: Log is configured to *not* fsync() on all Append() calls
I20260929 19:04:51.996204 21778 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cdc5863482cf4c88a6e6212fc5e82604: No bootstrap required, opened a new log
I20260929 19:04:51.998255 21778 raft_consensus.cc:348] T 00000000000000000000000000000000 P cdc5863482cf4c88a6e6212fc5e82604 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdc5863482cf4c88a6e6212fc5e82604" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41171 } }
I20260929 19:04:51.998409 21778 raft_consensus.cc:374] T 00000000000000000000000000000000 P cdc5863482cf4c88a6e6212fc5e82604 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:04:51.998432 21778 raft_consensus.cc:729] T 00000000000000000000000000000000 P cdc5863482cf4c88a6e6212fc5e82604 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdc5863482cf4c88a6e6212fc5e82604, State: Initialized, Role: FOLLOWER
I20260929 19:04:51.998576 21778 consensus_queue.cc:260] T 00000000000000000000000000000000 P cdc5863482cf4c88a6e6212fc5e82604 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdc5863482cf4c88a6e6212fc5e82604" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41171 } }
I20260929 19:04:51.998644 21778 raft_consensus.cc:388] T 00000000000000000000000000000000 P cdc5863482cf4c88a6e6212fc5e82604 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 19:04:51.998665 21778 raft_consensus.cc:482] T 00000000000000000000000000000000 P cdc5863482cf4c88a6e6212fc5e82604 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 19:04:51.998729 21778 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cdc5863482cf4c88a6e6212fc5e82604 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:04:51.999433 21778 raft_consensus.cc:504] T 00000000000000000000000000000000 P cdc5863482cf4c88a6e6212fc5e82604 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdc5863482cf4c88a6e6212fc5e82604" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41171 } }
I20260929 19:04:51.999548 21778 leader_election.cc:302] T 00000000000000000000000000000000 P cdc5863482cf4c88a6e6212fc5e82604 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cdc5863482cf4c88a6e6212fc5e82604; no voters:
I20260929 19:04:51.999754 21778 leader_election.cc:288] T 00000000000000000000000000000000 P cdc5863482cf4c88a6e6212fc5e82604 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 19:04:51.999779 21783 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cdc5863482cf4c88a6e6212fc5e82604 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:04:51.999882 21783 raft_consensus.cc:686] T 00000000000000000000000000000000 P cdc5863482cf4c88a6e6212fc5e82604 [term 1 LEADER]: Becoming Leader. State: Replica: cdc5863482cf4c88a6e6212fc5e82604, State: Running, Role: LEADER
I20260929 19:04:51.999987 21778 sys_catalog.cc:564] T 00000000000000000000000000000000 P cdc5863482cf4c88a6e6212fc5e82604 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 19:04:52.000016 21783 consensus_queue.cc:237] T 00000000000000000000000000000000 P cdc5863482cf4c88a6e6212fc5e82604 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdc5863482cf4c88a6e6212fc5e82604" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41171 } }
I20260929 19:04:52.000370 21783 sys_catalog.cc:455] T 00000000000000000000000000000000 P cdc5863482cf4c88a6e6212fc5e82604 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cdc5863482cf4c88a6e6212fc5e82604. Latest consensus state: current_term: 1 leader_uuid: "cdc5863482cf4c88a6e6212fc5e82604" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdc5863482cf4c88a6e6212fc5e82604" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41171 } } }
I20260929 19:04:52.000591 21783 sys_catalog.cc:458] T 00000000000000000000000000000000 P cdc5863482cf4c88a6e6212fc5e82604 [sys.catalog]: This master's current role is: LEADER
I20260929 19:04:52.000382 21784 sys_catalog.cc:455] T 00000000000000000000000000000000 P cdc5863482cf4c88a6e6212fc5e82604 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cdc5863482cf4c88a6e6212fc5e82604" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdc5863482cf4c88a6e6212fc5e82604" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41171 } } }
I20260929 19:04:52.000691 21784 sys_catalog.cc:458] T 00000000000000000000000000000000 P cdc5863482cf4c88a6e6212fc5e82604 [sys.catalog]: This master's current role is: LEADER
I20260929 19:04:52.000382 21781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:51.995234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.111:40671 (local address 127.0.0.1:41171)
0929 19:04:51.995406 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0929 19:04:51.995412 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 19:04:51.995439 (+ 27us) server_negotiation.cc:407] Connection header received
0929 19:04:51.995863 (+ 424us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:04:51.995869 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:04:51.995914 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:04:51.996030 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:04:51.996281 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:51.997138 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:51.998285 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:51.998864 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:51.998941 (+ 77us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:04:51.999593 (+ 652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:04:51.999597 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:04:51.999597 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 19:04:51.999782 (+ 185us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:04:51.999834 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 19:04:51.999849 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:04:52.000125 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0929 19:04:52.000163 (+ 38us) server_negotiation.cc:299] Negotiation successful
0929 19:04:52.000187 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":54,"threads_started":1}
I20260929 19:04:52.001085 21792 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 19:04:52.001693 21792 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 19:04:52.002064 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 19:04:52.003146 21792 catalog_manager.cc:1347] Generated new cluster ID: 6d0cd2ce41ed4f8a92d61dc2f1fb1dd7
I20260929 19:04:52.003189 21792 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 19:04:52.008107 21781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:52.004892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.111:49915 (local address 127.0.0.1:41171)
0929 19:04:52.004940 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0929 19:04:52.004942 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:04:52.004948 (+ 6us) server_negotiation.cc:407] Connection header received
0929 19:04:52.004964 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:04:52.004965 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:04:52.004972 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:04:52.005014 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:04:52.005186 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:52.005880 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:52.006839 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:52.007145 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:52.007224 (+ 79us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:04:52.007864 (+ 640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:04:52.007865 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:04:52.007865 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 19:04:52.007869 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:04:52.007885 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 19:04:52.007889 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:04:52.007952 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0929 19:04:52.008030 (+ 78us) server_negotiation.cc:299] Negotiation successful
0929 19:04:52.008039 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260929 19:04:52.025089 21792 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 19:04:52.025576 21792 catalog_manager.cc:1504] Loading token signing keys...
I20260929 19:04:52.030203 21792 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cdc5863482cf4c88a6e6212fc5e82604: Generated new TSK 0
I20260929 19:04:52.030396 21792 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 19:04:52.031845 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUbnLHE/build/release/bin/kudu with pid 21710
W20260929 19:04:52.035368 21695 connection.cc:535] client connection to 127.0.0.1:41171 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41171): Connection reset by peer (error 104)
2026-09-29T19:04:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1250 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790708447806542-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 19:04:52 dist-test-slave-lqkv krb5kdc[21809](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 19:04:52 dist-test-slave-lqkv krb5kdc[21809](info): set up 2 sockets
Sep 29 19:04:52 dist-test-slave-lqkv krb5kdc[21809](info): commencing operation
krb5kdc: starting...
W20260929 19:04:53.040951 392 mini_kdc.cc:121] Time spent starting KDC: real 0.992s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790708447806542-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790708447806542-392-0/krb5kdc/test-user.keytab.
Sep 29 19:04:53 dist-test-slave-lqkv krb5kdc[21809](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708693, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T19:04:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T19:04:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790708447806542-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790708447806542-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 19:04:53.149452 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUbnLHE/build/release/bin/kudu
/tmp/dist-test-taskUbnLHE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790708447806542-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790708447806542-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790708447806542-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790708447806542-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39831
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790708447806542-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39831
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790708447806542-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790708447806542-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790708447806542-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:04:53.228009 21822 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:04:53.228225 21822 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:04:53.228263 21822 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:04:53.229468 21822 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 19:04:53.229530 21822 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:04:53.229557 21822 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:04:53.229585 21822 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 19:04:53.229609 21822 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 19:04:53.231034 21822 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790708447806542-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790708447806542-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39831
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790708447806542-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39831
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790708447806542-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21822
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790708447806542-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 18:09:13 UTC on ccycloud.quasar-hpkalf.root.comops.site
build id 4680913
I20260929 19:04:53.231295 21822 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:04:53.231509 21822 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:04:53.236119 21828 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:04:53.236191 21827 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:04:53.236179 21830 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:04:53.236328 21822 server_base.cc:1034] running on GCE node
I20260929 19:04:53.236569 21822 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:04:53.236891 21822 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:04:53.238078 21822 hybrid_clock.cc:648] HybridClock initialized: now 1790708693238065 us; error 24 us; skew 500 ppm
Sep 29 19:04:53 dist-test-slave-lqkv krb5kdc[21809](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708693, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:04:53.240419 21822 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 19:04:53.240950 21822 webserver.cc:460] Webserver started at http://127.0.0.1:37311/ using document root <none> and password file <none>
I20260929 19:04:53.241191 21822 fs_manager.cc:359] Metadata directory not provided
I20260929 19:04:53.241256 21822 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:04:53.241387 21822 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:04:53.242368 21822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790708447806542-392-0/minicluster-data/master-0/data/instance:
uuid: "76e5107edb774d47b23986b148e76c78"
format_stamp: "Formatted at 2026-09-29 19:04:53 on dist-test-slave-lqkv"
server_key: "97eb8176a31ceeab42676223823b9136"
server_key_iv: "4eac848b16b201aa06e6eb403aebecea"
server_key_version: "encryptionkey@0"
I20260929 19:04:53.242714 21822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790708447806542-392-0/minicluster-data/master-0/wal/instance:
uuid: "76e5107edb774d47b23986b148e76c78"
format_stamp: "Formatted at 2026-09-29 19:04:53 on dist-test-slave-lqkv"
server_key: "97eb8176a31ceeab42676223823b9136"
server_key_iv: "4eac848b16b201aa06e6eb403aebecea"
server_key_version: "encryptionkey@0"
I20260929 19:04:53.243970 21822 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 19:04:53.244598 21837 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:04:53.244764 21822 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 19:04:53.244875 21822 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790708447806542-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790708447806542-392-0/minicluster-data/master-0/wal
uuid: "76e5107edb774d47b23986b148e76c78"
format_stamp: "Formatted at 2026-09-29 19:04:53 on dist-test-slave-lqkv"
server_key: "97eb8176a31ceeab42676223823b9136"
server_key_iv: "4eac848b16b201aa06e6eb403aebecea"
server_key_version: "encryptionkey@0"
I20260929 19:04:53.244964 21822 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790708447806542-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790708447806542-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790708447806542-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:04:53.248023 21822 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:04:53.249787 21822 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:04:53.249951 21822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:04:53.254065 21822 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39831
I20260929 19:04:53.254107 21889 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39831 every 8 connection(s)
I20260929 19:04:53.254421 21822 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790708447806542-392-0/minicluster-data/master-0/data/info.pb
I20260929 19:04:53.255266 21890 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:04:53.257373 21890 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76e5107edb774d47b23986b148e76c78: Bootstrap starting.
I20260929 19:04:53.257854 21890 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 76e5107edb774d47b23986b148e76c78: Neither blocks nor log segments found. Creating new log.
I20260929 19:04:53.258069 21890 log.cc:824] T 00000000000000000000000000000000 P 76e5107edb774d47b23986b148e76c78: Log is configured to *not* fsync() on all Append() calls
I20260929 19:04:53.258654 21890 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76e5107edb774d47b23986b148e76c78: No bootstrap required, opened a new log
I20260929 19:04:53.260115 21890 raft_consensus.cc:348] T 00000000000000000000000000000000 P 76e5107edb774d47b23986b148e76c78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76e5107edb774d47b23986b148e76c78" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39831 } }
I20260929 19:04:53.260234 21890 raft_consensus.cc:374] T 00000000000000000000000000000000 P 76e5107edb774d47b23986b148e76c78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:04:53.260255 21890 raft_consensus.cc:729] T 00000000000000000000000000000000 P 76e5107edb774d47b23986b148e76c78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76e5107edb774d47b23986b148e76c78, State: Initialized, Role: FOLLOWER
I20260929 19:04:53.260380 21890 consensus_queue.cc:260] T 00000000000000000000000000000000 P 76e5107edb774d47b23986b148e76c78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76e5107edb774d47b23986b148e76c78" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39831 } }
I20260929 19:04:53.260466 21890 raft_consensus.cc:388] T 00000000000000000000000000000000 P 76e5107edb774d47b23986b148e76c78 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 19:04:53.260490 21890 raft_consensus.cc:482] T 00000000000000000000000000000000 P 76e5107edb774d47b23986b148e76c78 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 19:04:53.260557 21890 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 76e5107edb774d47b23986b148e76c78 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:04:53.261116 21890 raft_consensus.cc:504] T 00000000000000000000000000000000 P 76e5107edb774d47b23986b148e76c78 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76e5107edb774d47b23986b148e76c78" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39831 } }
I20260929 19:04:53.261196 21890 leader_election.cc:302] T 00000000000000000000000000000000 P 76e5107edb774d47b23986b148e76c78 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 76e5107edb774d47b23986b148e76c78; no voters:
I20260929 19:04:53.261303 21890 leader_election.cc:288] T 00000000000000000000000000000000 P 76e5107edb774d47b23986b148e76c78 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 19:04:53.261358 21893 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 76e5107edb774d47b23986b148e76c78 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:04:53.261540 21893 raft_consensus.cc:686] T 00000000000000000000000000000000 P 76e5107edb774d47b23986b148e76c78 [term 1 LEADER]: Becoming Leader. State: Replica: 76e5107edb774d47b23986b148e76c78, State: Running, Role: LEADER
I20260929 19:04:53.261546 21890 sys_catalog.cc:564] T 00000000000000000000000000000000 P 76e5107edb774d47b23986b148e76c78 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 19:04:53.261664 21893 consensus_queue.cc:237] T 00000000000000000000000000000000 P 76e5107edb774d47b23986b148e76c78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76e5107edb774d47b23986b148e76c78" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39831 } }
I20260929 19:04:53.262228 21895 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76e5107edb774d47b23986b148e76c78 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 76e5107edb774d47b23986b148e76c78. Latest consensus state: current_term: 1 leader_uuid: "76e5107edb774d47b23986b148e76c78" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76e5107edb774d47b23986b148e76c78" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39831 } } }
I20260929 19:04:53.262320 21895 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76e5107edb774d47b23986b148e76c78 [sys.catalog]: This master's current role is: LEADER
I20260929 19:04:53.262552 21894 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76e5107edb774d47b23986b148e76c78 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "76e5107edb774d47b23986b148e76c78" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76e5107edb774d47b23986b148e76c78" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39831 } } }
I20260929 19:04:53.262600 21902 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 19:04:53.262684 21894 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76e5107edb774d47b23986b148e76c78 [sys.catalog]: This master's current role is: LEADER
I20260929 19:04:53.263155 21902 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 19:04:53.263504 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUbnLHE/build/release/bin/kudu as pid 21822
I20260929 19:04:53.263644 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790708447806542-392-0/minicluster-data/master-0/wal/instance
I20260929 19:04:53.263842 392 external_mini_cluster.cc:1428] Setting key bdc1ab5c8936c481684d4809a811bb1c
I20260929 19:04:53.264552 21902 catalog_manager.cc:1347] Generated new cluster ID: c6fd613ce522481085d6deac4482a446
I20260929 19:04:53.264601 21902 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 19:04:53.266305 21913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:53.264530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.111:45591 (local address 127.0.0.1:39831)
0929 19:04:53.264924 (+ 394us) server_negotiation.cc:206] Beginning negotiation
0929 19:04:53.264932 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 19:04:53.264961 (+ 29us) server_negotiation.cc:407] Connection header received
0929 19:04:53.265062 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:04:53.265069 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:04:53.265106 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:04:53.265235 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:04:53.265916 (+ 681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:04:53.265919 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:04:53.265919 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 19:04:53.265922 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:04:53.265955 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 19:04:53.265968 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:04:53.266025 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0929 19:04:53.266115 (+ 90us) server_negotiation.cc:299] Negotiation successful
0929 19:04:53.266151 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":63,"threads_started":1}
I20260929 19:04:53.267141 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 19:04:53.272195 21902 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 19:04:53.272341 21902 catalog_manager.cc:1504] Loading token signing keys...
I20260929 19:04:53.273931 21913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:53.268593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.111:32831 (local address 127.0.0.1:39831)
0929 19:04:53.268623 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0929 19:04:53.268625 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:04:53.272976 (+ 4351us) server_negotiation.cc:407] Connection header received
0929 19:04:53.273061 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:04:53.273062 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:04:53.273066 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:04:53.273109 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:04:53.273705 (+ 596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:04:53.273707 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:04:53.273707 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 19:04:53.273712 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:04:53.273724 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 19:04:53.273728 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:04:53.273784 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0929 19:04:53.273839 (+ 55us) server_negotiation.cc:299] Negotiation successful
0929 19:04:53.273852 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260929 19:04:53.278427 21902 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 76e5107edb774d47b23986b148e76c78: Generated new TSK 0
I20260929 19:04:53.278587 21902 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 19:04:53.285300 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUbnLHE/build/release/bin/kudu with pid 21822
2026-09-29T19:04:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1254 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790708447806542-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 19:04:53 dist-test-slave-lqkv krb5kdc[21921](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 19:04:53 dist-test-slave-lqkv krb5kdc[21921](info): set up 2 sockets
Sep 29 19:04:53 dist-test-slave-lqkv krb5kdc[21921](info): commencing operation
krb5kdc: starting...
W20260929 19:04:54.304399 392 mini_kdc.cc:121] Time spent starting KDC: real 1.002s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790708447806542-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790708447806542-392-0/krb5kdc/test-user.keytab.
Sep 29 19:04:54 dist-test-slave-lqkv krb5kdc[21921](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708694, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T19:04:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T19:04:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790708447806542-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790708447806542-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 19:04:54.409687 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUbnLHE/build/release/bin/kudu
/tmp/dist-test-taskUbnLHE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790708447806542-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790708447806542-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790708447806542-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790708447806542-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39765
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790708447806542-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39765
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790708447806542-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790708447806542-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790708447806542-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:04:54.487453 21934 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:04:54.487633 21934 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:04:54.487654 21934 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:04:54.488695 21934 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 19:04:54.488726 21934 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:04:54.488740 21934 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:04:54.488752 21934 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 19:04:54.488765 21934 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 19:04:54.490108 21934 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790708447806542-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790708447806542-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39765
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790708447806542-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39765
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790708447806542-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21934
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790708447806542-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 18:09:13 UTC on ccycloud.quasar-hpkalf.root.comops.site
build id 4680913
I20260929 19:04:54.490273 21934 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:04:54.490422 21934 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:04:54.494771 21942 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:04:54.494791 21939 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:04:54.494872 21940 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:04:54.494985 21934 server_base.cc:1034] running on GCE node
I20260929 19:04:54.495240 21934 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:04:54.495507 21934 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:04:54.496659 21934 hybrid_clock.cc:648] HybridClock initialized: now 1790708694496652 us; error 36 us; skew 500 ppm
Sep 29 19:04:54 dist-test-slave-lqkv krb5kdc[21921](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708694, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:04:54.499043 21934 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 19:04:54.499451 21934 webserver.cc:460] Webserver started at http://127.0.98.62:37185/ using document root <none> and password file <none>
I20260929 19:04:54.499624 21934 fs_manager.cc:359] Metadata directory not provided
I20260929 19:04:54.499679 21934 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:04:54.499759 21934 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:04:54.500658 21934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790708447806542-392-0/minicluster-data/master-0/data/instance:
uuid: "a650491204c8487d9910bb44e4161f2b"
format_stamp: "Formatted at 2026-09-29 19:04:54 on dist-test-slave-lqkv"
server_key: "dda287149f292535d4f609ac977b42c9"
server_key_iv: "68a46551abd25e26ddcd9bc52345f730"
server_key_version: "encryptionkey@0"
I20260929 19:04:54.500993 21934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790708447806542-392-0/minicluster-data/master-0/wal/instance:
uuid: "a650491204c8487d9910bb44e4161f2b"
format_stamp: "Formatted at 2026-09-29 19:04:54 on dist-test-slave-lqkv"
server_key: "dda287149f292535d4f609ac977b42c9"
server_key_iv: "68a46551abd25e26ddcd9bc52345f730"
server_key_version: "encryptionkey@0"
I20260929 19:04:54.502153 21934 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260929 19:04:54.502765 21949 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:04:54.502928 21934 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 19:04:54.503016 21934 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790708447806542-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790708447806542-392-0/minicluster-data/master-0/wal
uuid: "a650491204c8487d9910bb44e4161f2b"
format_stamp: "Formatted at 2026-09-29 19:04:54 on dist-test-slave-lqkv"
server_key: "dda287149f292535d4f609ac977b42c9"
server_key_iv: "68a46551abd25e26ddcd9bc52345f730"
server_key_version: "encryptionkey@0"
I20260929 19:04:54.503080 21934 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790708447806542-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790708447806542-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790708447806542-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:04:54.529961 21934 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:04:54.531527 21934 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:04:54.531666 21934 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:04:54.535292 21934 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39765
I20260929 19:04:54.535319 22001 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39765 every 8 connection(s)
I20260929 19:04:54.535609 21934 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790708447806542-392-0/minicluster-data/master-0/data/info.pb
I20260929 19:04:54.536331 22002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:04:54.538386 22002 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a650491204c8487d9910bb44e4161f2b: Bootstrap starting.
I20260929 19:04:54.538910 22002 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a650491204c8487d9910bb44e4161f2b: Neither blocks nor log segments found. Creating new log.
I20260929 19:04:54.539177 22002 log.cc:824] T 00000000000000000000000000000000 P a650491204c8487d9910bb44e4161f2b: Log is configured to *not* fsync() on all Append() calls
I20260929 19:04:54.539788 22002 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a650491204c8487d9910bb44e4161f2b: No bootstrap required, opened a new log
I20260929 19:04:54.541364 22002 raft_consensus.cc:348] T 00000000000000000000000000000000 P a650491204c8487d9910bb44e4161f2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a650491204c8487d9910bb44e4161f2b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39765 } }
I20260929 19:04:54.541488 22002 raft_consensus.cc:374] T 00000000000000000000000000000000 P a650491204c8487d9910bb44e4161f2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:04:54.541530 22002 raft_consensus.cc:729] T 00000000000000000000000000000000 P a650491204c8487d9910bb44e4161f2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a650491204c8487d9910bb44e4161f2b, State: Initialized, Role: FOLLOWER
I20260929 19:04:54.541679 22002 consensus_queue.cc:260] T 00000000000000000000000000000000 P a650491204c8487d9910bb44e4161f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a650491204c8487d9910bb44e4161f2b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39765 } }
I20260929 19:04:54.541739 22002 raft_consensus.cc:388] T 00000000000000000000000000000000 P a650491204c8487d9910bb44e4161f2b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 19:04:54.541759 22002 raft_consensus.cc:482] T 00000000000000000000000000000000 P a650491204c8487d9910bb44e4161f2b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 19:04:54.541821 22002 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a650491204c8487d9910bb44e4161f2b [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:04:54.542315 22002 raft_consensus.cc:504] T 00000000000000000000000000000000 P a650491204c8487d9910bb44e4161f2b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a650491204c8487d9910bb44e4161f2b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39765 } }
I20260929 19:04:54.542445 22002 leader_election.cc:302] T 00000000000000000000000000000000 P a650491204c8487d9910bb44e4161f2b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a650491204c8487d9910bb44e4161f2b; no voters:
I20260929 19:04:54.542627 22002 leader_election.cc:288] T 00000000000000000000000000000000 P a650491204c8487d9910bb44e4161f2b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 19:04:54.542657 22005 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a650491204c8487d9910bb44e4161f2b [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:04:54.542810 22005 raft_consensus.cc:686] T 00000000000000000000000000000000 P a650491204c8487d9910bb44e4161f2b [term 1 LEADER]: Becoming Leader. State: Replica: a650491204c8487d9910bb44e4161f2b, State: Running, Role: LEADER
I20260929 19:04:54.542860 22002 sys_catalog.cc:564] T 00000000000000000000000000000000 P a650491204c8487d9910bb44e4161f2b [sys.catalog]: configured and running, proceeding with master startup.
I20260929 19:04:54.542943 22005 consensus_queue.cc:237] T 00000000000000000000000000000000 P a650491204c8487d9910bb44e4161f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a650491204c8487d9910bb44e4161f2b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39765 } }
I20260929 19:04:54.543772 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUbnLHE/build/release/bin/kudu as pid 21934
I20260929 19:04:54.543905 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790708447806542-392-0/minicluster-data/master-0/wal/instance
I20260929 19:04:54.544180 392 external_mini_cluster.cc:1428] Setting key f788ad3eb5030f1ffedc2386bd5168e3
I20260929 19:04:54.544296 22007 sys_catalog.cc:455] T 00000000000000000000000000000000 P a650491204c8487d9910bb44e4161f2b [sys.catalog]: SysCatalogTable state changed. Reason: New leader a650491204c8487d9910bb44e4161f2b. Latest consensus state: current_term: 1 leader_uuid: "a650491204c8487d9910bb44e4161f2b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a650491204c8487d9910bb44e4161f2b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39765 } } }
I20260929 19:04:54.544389 22007 sys_catalog.cc:458] T 00000000000000000000000000000000 P a650491204c8487d9910bb44e4161f2b [sys.catalog]: This master's current role is: LEADER
I20260929 19:04:54.544600 22015 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 19:04:54.544907 22006 sys_catalog.cc:455] T 00000000000000000000000000000000 P a650491204c8487d9910bb44e4161f2b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a650491204c8487d9910bb44e4161f2b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a650491204c8487d9910bb44e4161f2b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39765 } } }
I20260929 19:04:54.545030 22006 sys_catalog.cc:458] T 00000000000000000000000000000000 P a650491204c8487d9910bb44e4161f2b [sys.catalog]: This master's current role is: LEADER
I20260929 19:04:54.545147 22015 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 19:04:54.546682 22015 catalog_manager.cc:1347] Generated new cluster ID: 2ca0b7ca57f448d88fc4107897472b34
I20260929 19:04:54.546727 22015 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 29 19:04:54 dist-test-slave-lqkv krb5kdc[21921](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708694, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 19:04:54.556357 22022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:54.544879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34883 (local address 127.0.98.62:39765)
0929 19:04:54.545053 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0929 19:04:54.545063 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 19:04:54.545211 (+ 148us) server_negotiation.cc:407] Connection header received
0929 19:04:54.545381 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:04:54.545389 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:04:54.545454 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:04:54.545589 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:04:54.546503 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:54.547997 (+ 1494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:54.549143 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:54.549744 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:54.549855 (+ 111us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:04:54.552632 (+ 2777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:04:54.552645 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:04:54.552645 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:04:54.552659 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:04:54.554646 (+ 1987us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:04:54.555064 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:04:54.555066 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:04:54.555067 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:04:54.555194 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:04:54.555415 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:04:54.555416 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:04:54.555416 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:04:54.555705 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:04:54.555874 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0929 19:04:54.555955 (+ 81us) server_negotiation.cc:299] Negotiation successful
0929 19:04:54.556156 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":67,"threads_started":1}
I20260929 19:04:54.557523 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 19:04:54.557982 22015 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 19:04:54.558398 22015 catalog_manager.cc:1504] Loading token signing keys...
I20260929 19:04:54.562762 22015 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a650491204c8487d9910bb44e4161f2b: Generated new TSK 0
I20260929 19:04:54.562915 22015 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 19:04:54.566018 22022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:54.558904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48743 (local address 127.0.98.62:39765)
0929 19:04:54.558939 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0929 19:04:54.558941 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:04:54.559085 (+ 144us) server_negotiation.cc:407] Connection header received
0929 19:04:54.559174 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:04:54.559175 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:04:54.559184 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:04:54.559211 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:04:54.559921 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:54.560681 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:54.561732 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:54.562055 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:54.562092 (+ 37us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:04:54.563446 (+ 1354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:04:54.563447 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:04:54.563447 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:04:54.563454 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:04:54.564991 (+ 1537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:04:54.565323 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:04:54.565324 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:04:54.565324 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:04:54.565380 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:04:54.565573 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:04:54.565574 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:04:54.565574 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:04:54.565690 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:04:54.565786 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0929 19:04:54.565873 (+ 87us) server_negotiation.cc:299] Negotiation successful
0929 19:04:54.565934 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260929 19:04:54.567622 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUbnLHE/build/release/bin/kudu with pid 21934
W20260929 19:04:54.571219 21919 connection.cc:535] client connection to 127.0.98.62:39765 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:39765): Connection reset by peer (error 104)
2026-09-29T19:04:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1281 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790708447806542-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 19:04:54 dist-test-slave-lqkv krb5kdc[22033](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 19:04:54 dist-test-slave-lqkv krb5kdc[22033](info): set up 2 sockets
Sep 29 19:04:54 dist-test-slave-lqkv krb5kdc[22033](info): commencing operation
krb5kdc: starting...
W20260929 19:04:55.568781 392 mini_kdc.cc:121] Time spent starting KDC: real 0.984s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790708447806542-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790708447806542-392-0/krb5kdc/test-user.keytab.
Sep 29 19:04:55 dist-test-slave-lqkv krb5kdc[22033](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708695, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T19:04:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T19:04:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790708447806542-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790708447806542-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 19:04:55.679255 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUbnLHE/build/release/bin/kudu
/tmp/dist-test-taskUbnLHE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790708447806542-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790708447806542-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790708447806542-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790708447806542-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33801
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790708447806542-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33801
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790708447806542-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790708447806542-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790708447806542-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:04:55.754851 22046 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:04:55.755033 22046 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:04:55.755054 22046 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:04:55.756107 22046 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 19:04:55.756146 22046 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:04:55.756160 22046 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:04:55.756172 22046 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 19:04:55.756217 22046 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 19:04:55.757589 22046 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790708447806542-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790708447806542-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33801
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790708447806542-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33801
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790708447806542-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22046
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790708447806542-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 18:09:13 UTC on ccycloud.quasar-hpkalf.root.comops.site
build id 4680913
I20260929 19:04:55.757745 22046 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:04:55.757936 22046 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:04:55.762424 22051 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:04:55.762516 22052 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:04:55.762512 22054 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:04:55.762601 22046 server_base.cc:1034] running on GCE node
I20260929 19:04:55.762914 22046 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:04:55.763185 22046 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:04:55.764336 22046 hybrid_clock.cc:648] HybridClock initialized: now 1790708695764329 us; error 35 us; skew 500 ppm
Sep 29 19:04:55 dist-test-slave-lqkv krb5kdc[22033](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708695, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:04:55.766494 22046 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 19:04:55.766906 22046 webserver.cc:460] Webserver started at http://127.0.98.62:40977/ using document root <none> and password file <none>
I20260929 19:04:55.767112 22046 fs_manager.cc:359] Metadata directory not provided
I20260929 19:04:55.767172 22046 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:04:55.767315 22046 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:04:55.768260 22046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790708447806542-392-0/minicluster-data/master-0/data/instance:
uuid: "10af3a77c6bb4f57af329abc6ee62286"
format_stamp: "Formatted at 2026-09-29 19:04:55 on dist-test-slave-lqkv"
server_key: "e5330a95ced60f09707c49439563f21d"
server_key_iv: "7bb06dfce0dd9f3f22edf506402462e3"
server_key_version: "encryptionkey@0"
I20260929 19:04:55.768591 22046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790708447806542-392-0/minicluster-data/master-0/wal/instance:
uuid: "10af3a77c6bb4f57af329abc6ee62286"
format_stamp: "Formatted at 2026-09-29 19:04:55 on dist-test-slave-lqkv"
server_key: "e5330a95ced60f09707c49439563f21d"
server_key_iv: "7bb06dfce0dd9f3f22edf506402462e3"
server_key_version: "encryptionkey@0"
I20260929 19:04:55.769852 22046 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260929 19:04:55.770502 22061 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:04:55.770654 22046 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 19:04:55.770740 22046 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790708447806542-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790708447806542-392-0/minicluster-data/master-0/wal
uuid: "10af3a77c6bb4f57af329abc6ee62286"
format_stamp: "Formatted at 2026-09-29 19:04:55 on dist-test-slave-lqkv"
server_key: "e5330a95ced60f09707c49439563f21d"
server_key_iv: "7bb06dfce0dd9f3f22edf506402462e3"
server_key_version: "encryptionkey@0"
I20260929 19:04:55.770812 22046 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790708447806542-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790708447806542-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790708447806542-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:04:55.785930 22046 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:04:55.787464 22046 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:04:55.787606 22046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:04:55.791386 22046 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33801
I20260929 19:04:55.791419 22113 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33801 every 8 connection(s)
I20260929 19:04:55.791718 22046 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790708447806542-392-0/minicluster-data/master-0/data/info.pb
I20260929 19:04:55.792464 22114 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:04:55.793045 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUbnLHE/build/release/bin/kudu as pid 22046
I20260929 19:04:55.793170 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790708447806542-392-0/minicluster-data/master-0/wal/instance
I20260929 19:04:55.793386 392 external_mini_cluster.cc:1428] Setting key cf1920bfe4fc25235a566369bf49d837
I20260929 19:04:55.795342 22114 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10af3a77c6bb4f57af329abc6ee62286: Bootstrap starting.
I20260929 19:04:55.795966 22114 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 10af3a77c6bb4f57af329abc6ee62286: Neither blocks nor log segments found. Creating new log.
I20260929 19:04:55.796236 22114 log.cc:824] T 00000000000000000000000000000000 P 10af3a77c6bb4f57af329abc6ee62286: Log is configured to *not* fsync() on all Append() calls
I20260929 19:04:55.796888 22114 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 10af3a77c6bb4f57af329abc6ee62286: No bootstrap required, opened a new log
I20260929 19:04:55.798758 22117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:55.793824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38349 (local address 127.0.98.62:33801)
0929 19:04:55.794000 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0929 19:04:55.794009 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 19:04:55.794039 (+ 30us) server_negotiation.cc:407] Connection header received
0929 19:04:55.794561 (+ 522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:04:55.794569 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:04:55.794629 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:04:55.794768 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:04:55.794907 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:55.796215 (+ 1308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:55.797004 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:55.797404 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:55.797509 (+ 105us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:04:55.798190 (+ 681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:04:55.798193 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:04:55.798193 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 19:04:55.798333 (+ 140us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:04:55.798358 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 19:04:55.798372 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:04:55.798476 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 19:04:55.798581 (+ 105us) server_negotiation.cc:299] Negotiation successful
0929 19:04:55.798608 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":73,"threads_started":1}
I20260929 19:04:55.798872 22114 raft_consensus.cc:348] T 00000000000000000000000000000000 P 10af3a77c6bb4f57af329abc6ee62286 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10af3a77c6bb4f57af329abc6ee62286" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33801 } }
I20260929 19:04:55.799028 22114 raft_consensus.cc:374] T 00000000000000000000000000000000 P 10af3a77c6bb4f57af329abc6ee62286 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:04:55.799072 22114 raft_consensus.cc:729] T 00000000000000000000000000000000 P 10af3a77c6bb4f57af329abc6ee62286 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10af3a77c6bb4f57af329abc6ee62286, State: Initialized, Role: FOLLOWER
I20260929 19:04:55.799185 22114 consensus_queue.cc:260] T 00000000000000000000000000000000 P 10af3a77c6bb4f57af329abc6ee62286 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10af3a77c6bb4f57af329abc6ee62286" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33801 } }
I20260929 19:04:55.799261 22114 raft_consensus.cc:388] T 00000000000000000000000000000000 P 10af3a77c6bb4f57af329abc6ee62286 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 19:04:55.799293 22114 raft_consensus.cc:482] T 00000000000000000000000000000000 P 10af3a77c6bb4f57af329abc6ee62286 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 19:04:55.799338 22114 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 10af3a77c6bb4f57af329abc6ee62286 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:04:55.799604 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 19:04:55.799942 22114 raft_consensus.cc:504] T 00000000000000000000000000000000 P 10af3a77c6bb4f57af329abc6ee62286 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10af3a77c6bb4f57af329abc6ee62286" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33801 } }
I20260929 19:04:55.800042 22114 leader_election.cc:302] T 00000000000000000000000000000000 P 10af3a77c6bb4f57af329abc6ee62286 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 10af3a77c6bb4f57af329abc6ee62286; no voters:
I20260929 19:04:55.800185 22114 leader_election.cc:288] T 00000000000000000000000000000000 P 10af3a77c6bb4f57af329abc6ee62286 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 19:04:55.800230 22119 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 10af3a77c6bb4f57af329abc6ee62286 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:04:55.800381 22119 raft_consensus.cc:686] T 00000000000000000000000000000000 P 10af3a77c6bb4f57af329abc6ee62286 [term 1 LEADER]: Becoming Leader. State: Replica: 10af3a77c6bb4f57af329abc6ee62286, State: Running, Role: LEADER
I20260929 19:04:55.800416 22114 sys_catalog.cc:564] T 00000000000000000000000000000000 P 10af3a77c6bb4f57af329abc6ee62286 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 19:04:55.800535 22119 consensus_queue.cc:237] T 00000000000000000000000000000000 P 10af3a77c6bb4f57af329abc6ee62286 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10af3a77c6bb4f57af329abc6ee62286" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33801 } }
I20260929 19:04:55.800894 22121 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10af3a77c6bb4f57af329abc6ee62286 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "10af3a77c6bb4f57af329abc6ee62286" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10af3a77c6bb4f57af329abc6ee62286" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33801 } } }
I20260929 19:04:55.800976 22121 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10af3a77c6bb4f57af329abc6ee62286 [sys.catalog]: This master's current role is: LEADER
I20260929 19:04:55.801113 22121 sys_catalog.cc:455] T 00000000000000000000000000000000 P 10af3a77c6bb4f57af329abc6ee62286 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 10af3a77c6bb4f57af329abc6ee62286. Latest consensus state: current_term: 1 leader_uuid: "10af3a77c6bb4f57af329abc6ee62286" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "10af3a77c6bb4f57af329abc6ee62286" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33801 } } }
I20260929 19:04:55.801177 22121 sys_catalog.cc:458] T 00000000000000000000000000000000 P 10af3a77c6bb4f57af329abc6ee62286 [sys.catalog]: This master's current role is: LEADER
I20260929 19:04:55.801277 22131 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 19:04:55.801800 22131 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 19:04:55.803364 22131 catalog_manager.cc:1347] Generated new cluster ID: 7a46ba8aa6cb4ad59a24bcfad65da1ef
I20260929 19:04:55.803409 22131 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 19:04:55.804515 22117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:55.801474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47089 (local address 127.0.98.62:33801)
0929 19:04:55.801507 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0929 19:04:55.801509 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:04:55.801656 (+ 147us) server_negotiation.cc:407] Connection header received
0929 19:04:55.801753 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:04:55.801754 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:04:55.801763 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:04:55.801810 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:04:55.801963 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:55.802489 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:55.803403 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:55.803640 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:55.803714 (+ 74us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:04:55.804297 (+ 583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:04:55.804298 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:04:55.804298 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 19:04:55.804302 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:04:55.804313 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 19:04:55.804317 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:04:55.804367 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0929 19:04:55.804423 (+ 56us) server_negotiation.cc:299] Negotiation successful
0929 19:04:55.804431 (+ 8us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260929 19:04:55.815830 22131 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 19:04:55.816255 22131 catalog_manager.cc:1504] Loading token signing keys...
I20260929 19:04:55.831815 22131 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 10af3a77c6bb4f57af329abc6ee62286: Generated new TSK 0
I20260929 19:04:55.831972 22131 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 19:04:55.840287 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUbnLHE/build/release/bin/kudu with pid 22046
W20260929 19:04:55.844216 22031 connection.cc:535] client connection to 127.0.98.62:33801 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:33801): Connection reset by peer (error 104)
2026-09-29T19:04:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1272 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790708447806542-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 19:04:55 dist-test-slave-lqkv krb5kdc[22145](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 19:04:55 dist-test-slave-lqkv krb5kdc[22145](info): set up 2 sockets
Sep 29 19:04:55 dist-test-slave-lqkv krb5kdc[22145](info): commencing operation
krb5kdc: starting...
W20260929 19:04:56.868981 392 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790708447806542-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790708447806542-392-0/krb5kdc/test-user.keytab.
Sep 29 19:04:56 dist-test-slave-lqkv krb5kdc[22145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708696, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T19:04:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T19:04:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790708447806542-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790708447806542-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 19:04:56.983315 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUbnLHE/build/release/bin/kudu
/tmp/dist-test-taskUbnLHE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790708447806542-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790708447806542-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790708447806542-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790708447806542-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40397
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790708447806542-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40397
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790708447806542-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790708447806542-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790708447806542-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:04:57.065047 22158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:04:57.065281 22158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:04:57.065306 22158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:04:57.066448 22158 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 19:04:57.066488 22158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:04:57.066501 22158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:04:57.066521 22158 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 19:04:57.066533 22158 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 19:04:57.067979 22158 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790708447806542-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790708447806542-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40397
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790708447806542-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40397
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790708447806542-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790708447806542-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 18:09:13 UTC on ccycloud.quasar-hpkalf.root.comops.site
build id 4680913
I20260929 19:04:57.068188 22158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:04:57.068373 22158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:04:57.073380 22164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:04:57.073390 22163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:04:57.073421 22158 server_base.cc:1034] running on GCE node
W20260929 19:04:57.073395 22166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:04:57.073861 22158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:04:57.074172 22158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:04:57.075340 22158 hybrid_clock.cc:648] HybridClock initialized: now 1790708697075289 us; error 58 us; skew 500 ppm
Sep 29 19:04:57 dist-test-slave-lqkv krb5kdc[22145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708697, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:04:57.082144 22158 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 19:04:57.082662 22158 webserver.cc:460] Webserver started at http://127.0.98.62:41517/ using document root <none> and password file <none>
I20260929 19:04:57.082864 22158 fs_manager.cc:359] Metadata directory not provided
I20260929 19:04:57.082907 22158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:04:57.082999 22158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:04:57.084052 22158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790708447806542-392-0/minicluster-data/master-0/data/instance:
uuid: "d24d91b36465450abbf34b2f5fa65339"
format_stamp: "Formatted at 2026-09-29 19:04:57 on dist-test-slave-lqkv"
server_key: "48bae8630f8b3d9b86d25cd1bb180d4d"
server_key_iv: "9c0dc97d82c4d407be04063d4d0181f5"
server_key_version: "encryptionkey@0"
I20260929 19:04:57.084363 22158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790708447806542-392-0/minicluster-data/master-0/wal/instance:
uuid: "d24d91b36465450abbf34b2f5fa65339"
format_stamp: "Formatted at 2026-09-29 19:04:57 on dist-test-slave-lqkv"
server_key: "48bae8630f8b3d9b86d25cd1bb180d4d"
server_key_iv: "9c0dc97d82c4d407be04063d4d0181f5"
server_key_version: "encryptionkey@0"
I20260929 19:04:57.085778 22158 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 19:04:57.086632 22173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:04:57.086838 22158 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 19:04:57.086923 22158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790708447806542-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790708447806542-392-0/minicluster-data/master-0/wal
uuid: "d24d91b36465450abbf34b2f5fa65339"
format_stamp: "Formatted at 2026-09-29 19:04:57 on dist-test-slave-lqkv"
server_key: "48bae8630f8b3d9b86d25cd1bb180d4d"
server_key_iv: "9c0dc97d82c4d407be04063d4d0181f5"
server_key_version: "encryptionkey@0"
I20260929 19:04:57.086980 22158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790708447806542-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790708447806542-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790708447806542-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:04:57.090241 22158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:04:57.092072 22158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:04:57.092180 22158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:04:57.095834 22158 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40397
I20260929 19:04:57.095878 22225 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40397 every 8 connection(s)
I20260929 19:04:57.096205 22158 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790708447806542-392-0/minicluster-data/master-0/data/info.pb
I20260929 19:04:57.097333 22226 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:04:57.097910 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUbnLHE/build/release/bin/kudu as pid 22158
I20260929 19:04:57.098034 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790708447806542-392-0/minicluster-data/master-0/wal/instance
I20260929 19:04:57.098199 392 external_mini_cluster.cc:1428] Setting key 6290c24925a117b1acf876fb91322767
I20260929 19:04:57.100448 22226 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d24d91b36465450abbf34b2f5fa65339: Bootstrap starting.
I20260929 19:04:57.100623 22229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:57.098676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41515 (local address 127.0.98.62:40397)
0929 19:04:57.098826 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0929 19:04:57.098838 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 19:04:57.098866 (+ 28us) server_negotiation.cc:407] Connection header received
0929 19:04:57.099348 (+ 482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:04:57.099356 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:04:57.099381 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:04:57.099492 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:04:57.100130 (+ 638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:04:57.100134 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:04:57.100134 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 19:04:57.100137 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:04:57.100172 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 19:04:57.100182 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:04:57.100235 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0929 19:04:57.100322 (+ 87us) server_negotiation.cc:299] Negotiation successful
0929 19:04:57.100353 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":70,"threads_started":1}
I20260929 19:04:57.101179 22226 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d24d91b36465450abbf34b2f5fa65339: Neither blocks nor log segments found. Creating new log.
I20260929 19:04:57.101523 22226 log.cc:824] T 00000000000000000000000000000000 P d24d91b36465450abbf34b2f5fa65339: Log is configured to *not* fsync() on all Append() calls
I20260929 19:04:57.101537 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 19:04:57.102242 22226 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d24d91b36465450abbf34b2f5fa65339: No bootstrap required, opened a new log
I20260929 19:04:57.103911 22229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:57.102989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52593 (local address 127.0.98.62:40397)
0929 19:04:57.103027 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0929 19:04:57.103028 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 19:04:57.103100 (+ 72us) server_negotiation.cc:407] Connection header received
0929 19:04:57.103165 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:04:57.103166 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:04:57.103169 (+ 3us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:04:57.103212 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:04:57.103713 (+ 501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:04:57.103714 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:04:57.103715 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 19:04:57.103717 (+ 2us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:04:57.103727 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 19:04:57.103730 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:04:57.103766 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0929 19:04:57.103825 (+ 59us) server_negotiation.cc:299] Negotiation successful
0929 19:04:57.103835 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260929 19:04:57.104686 22226 raft_consensus.cc:348] T 00000000000000000000000000000000 P d24d91b36465450abbf34b2f5fa65339 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d24d91b36465450abbf34b2f5fa65339" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40397 } }
I20260929 19:04:57.104868 22226 raft_consensus.cc:374] T 00000000000000000000000000000000 P d24d91b36465450abbf34b2f5fa65339 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:04:57.104918 22226 raft_consensus.cc:729] T 00000000000000000000000000000000 P d24d91b36465450abbf34b2f5fa65339 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d24d91b36465450abbf34b2f5fa65339, State: Initialized, Role: FOLLOWER
I20260929 19:04:57.105049 22226 consensus_queue.cc:260] T 00000000000000000000000000000000 P d24d91b36465450abbf34b2f5fa65339 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d24d91b36465450abbf34b2f5fa65339" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40397 } }
I20260929 19:04:57.105139 22226 raft_consensus.cc:388] T 00000000000000000000000000000000 P d24d91b36465450abbf34b2f5fa65339 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 19:04:57.105183 22226 raft_consensus.cc:482] T 00000000000000000000000000000000 P d24d91b36465450abbf34b2f5fa65339 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 19:04:57.105233 22226 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d24d91b36465450abbf34b2f5fa65339 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:04:57.105813 22226 raft_consensus.cc:504] T 00000000000000000000000000000000 P d24d91b36465450abbf34b2f5fa65339 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d24d91b36465450abbf34b2f5fa65339" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40397 } }
I20260929 19:04:57.105926 22226 leader_election.cc:302] T 00000000000000000000000000000000 P d24d91b36465450abbf34b2f5fa65339 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d24d91b36465450abbf34b2f5fa65339; no voters:
I20260929 19:04:57.106117 22226 leader_election.cc:288] T 00000000000000000000000000000000 P d24d91b36465450abbf34b2f5fa65339 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 19:04:57.106153 22236 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d24d91b36465450abbf34b2f5fa65339 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:04:57.106319 22236 raft_consensus.cc:686] T 00000000000000000000000000000000 P d24d91b36465450abbf34b2f5fa65339 [term 1 LEADER]: Becoming Leader. State: Replica: d24d91b36465450abbf34b2f5fa65339, State: Running, Role: LEADER
I20260929 19:04:57.106359 22226 sys_catalog.cc:564] T 00000000000000000000000000000000 P d24d91b36465450abbf34b2f5fa65339 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 19:04:57.106472 22236 consensus_queue.cc:237] T 00000000000000000000000000000000 P d24d91b36465450abbf34b2f5fa65339 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d24d91b36465450abbf34b2f5fa65339" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40397 } }
I20260929 19:04:57.107591 22238 sys_catalog.cc:455] T 00000000000000000000000000000000 P d24d91b36465450abbf34b2f5fa65339 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d24d91b36465450abbf34b2f5fa65339. Latest consensus state: current_term: 1 leader_uuid: "d24d91b36465450abbf34b2f5fa65339" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d24d91b36465450abbf34b2f5fa65339" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40397 } } }
I20260929 19:04:57.107695 22238 sys_catalog.cc:458] T 00000000000000000000000000000000 P d24d91b36465450abbf34b2f5fa65339 [sys.catalog]: This master's current role is: LEADER
I20260929 19:04:57.107889 22237 sys_catalog.cc:455] T 00000000000000000000000000000000 P d24d91b36465450abbf34b2f5fa65339 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d24d91b36465450abbf34b2f5fa65339" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d24d91b36465450abbf34b2f5fa65339" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40397 } } }
I20260929 19:04:57.108062 22237 sys_catalog.cc:458] T 00000000000000000000000000000000 P d24d91b36465450abbf34b2f5fa65339 [sys.catalog]: This master's current role is: LEADER
W20260929 19:04:57.108247 22251 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d24d91b36465450abbf34b2f5fa65339: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 19:04:57.108304 22251 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 19:04:57.108350 22252 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 19:04:57.108481 22252 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 19:04:57.109841 22252 catalog_manager.cc:1347] Generated new cluster ID: 7120bb96530943f4b42505996e7ae1b8
I20260929 19:04:57.109891 22252 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 19:04:57.116607 22252 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 19:04:57.116765 22252 catalog_manager.cc:1504] Loading token signing keys...
I20260929 19:04:57.139910 22252 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d24d91b36465450abbf34b2f5fa65339: Generated new TSK 0
I20260929 19:04:57.140136 22252 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 19:04:57.152423 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUbnLHE/build/release/bin/kudu with pid 22158
2026-09-29T19:04:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1312 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790708447806542-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 19:04:57 dist-test-slave-lqkv krb5kdc[22257](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 19:04:57 dist-test-slave-lqkv krb5kdc[22257](info): set up 2 sockets
Sep 29 19:04:57 dist-test-slave-lqkv krb5kdc[22257](info): commencing operation
krb5kdc: starting...
W20260929 19:04:58.184906 392 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790708447806542-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790708447806542-392-0/krb5kdc/test-user.keytab.
Sep 29 19:04:58 dist-test-slave-lqkv krb5kdc[22257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708698, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T19:04:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T19:04:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790708447806542-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790708447806542-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 19:04:58.303946 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUbnLHE/build/release/bin/kudu
/tmp/dist-test-taskUbnLHE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790708447806542-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790708447806542-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790708447806542-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790708447806542-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38185
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790708447806542-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38185
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790708447806542-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790708447806542-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790708447806542-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:04:58.382974 22270 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:04:58.383216 22270 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:04:58.383255 22270 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:04:58.384392 22270 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 19:04:58.384462 22270 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:04:58.384514 22270 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:04:58.384562 22270 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 19:04:58.384585 22270 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 19:04:58.386077 22270 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790708447806542-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790708447806542-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38185
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790708447806542-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38185
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790708447806542-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22270
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790708447806542-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 18:09:13 UTC on ccycloud.quasar-hpkalf.root.comops.site
build id 4680913
I20260929 19:04:58.386348 22270 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:04:58.386579 22270 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:04:58.391768 22276 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:04:58.391780 22278 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:04:58.391764 22275 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:04:58.391835 22270 server_base.cc:1034] running on GCE node
I20260929 19:04:58.392262 22270 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:04:58.392554 22270 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:04:58.393688 22270 hybrid_clock.cc:648] HybridClock initialized: now 1790708698393653 us; error 54 us; skew 500 ppm
Sep 29 19:04:58 dist-test-slave-lqkv krb5kdc[22257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708698, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:04:58.396222 22270 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 19:04:58.396667 22270 webserver.cc:460] Webserver started at http://127.0.98.62:36173/ using document root <none> and password file <none>
I20260929 19:04:58.396932 22270 fs_manager.cc:359] Metadata directory not provided
I20260929 19:04:58.396979 22270 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:04:58.397114 22270 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:04:58.398169 22270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790708447806542-392-0/minicluster-data/master-0/data/instance:
uuid: "a228a78d489a42dd9cb3336b883ac964"
format_stamp: "Formatted at 2026-09-29 19:04:58 on dist-test-slave-lqkv"
server_key: "592732e7ca3e867d49f89b9b49cd1b6f"
server_key_iv: "ef5bd7ba9599765a61cfcbe6cb1c0891"
server_key_version: "encryptionkey@0"
I20260929 19:04:58.398517 22270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790708447806542-392-0/minicluster-data/master-0/wal/instance:
uuid: "a228a78d489a42dd9cb3336b883ac964"
format_stamp: "Formatted at 2026-09-29 19:04:58 on dist-test-slave-lqkv"
server_key: "592732e7ca3e867d49f89b9b49cd1b6f"
server_key_iv: "ef5bd7ba9599765a61cfcbe6cb1c0891"
server_key_version: "encryptionkey@0"
I20260929 19:04:58.399837 22270 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260929 19:04:58.400547 22285 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:04:58.400724 22270 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 19:04:58.400842 22270 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790708447806542-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790708447806542-392-0/minicluster-data/master-0/wal
uuid: "a228a78d489a42dd9cb3336b883ac964"
format_stamp: "Formatted at 2026-09-29 19:04:58 on dist-test-slave-lqkv"
server_key: "592732e7ca3e867d49f89b9b49cd1b6f"
server_key_iv: "ef5bd7ba9599765a61cfcbe6cb1c0891"
server_key_version: "encryptionkey@0"
I20260929 19:04:58.400923 22270 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790708447806542-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790708447806542-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790708447806542-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:04:58.426260 22270 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:04:58.428722 22270 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:04:58.428910 22270 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:04:58.432967 22270 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38185
I20260929 19:04:58.433010 22337 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38185 every 8 connection(s)
I20260929 19:04:58.433364 22270 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790708447806542-392-0/minicluster-data/master-0/data/info.pb
I20260929 19:04:58.434194 22338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:04:58.436360 22338 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a228a78d489a42dd9cb3336b883ac964: Bootstrap starting.
I20260929 19:04:58.436960 22338 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a228a78d489a42dd9cb3336b883ac964: Neither blocks nor log segments found. Creating new log.
I20260929 19:04:58.437196 22338 log.cc:824] T 00000000000000000000000000000000 P a228a78d489a42dd9cb3336b883ac964: Log is configured to *not* fsync() on all Append() calls
I20260929 19:04:58.437842 22338 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a228a78d489a42dd9cb3336b883ac964: No bootstrap required, opened a new log
I20260929 19:04:58.438724 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUbnLHE/build/release/bin/kudu as pid 22270
I20260929 19:04:58.438849 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790708447806542-392-0/minicluster-data/master-0/wal/instance
I20260929 19:04:58.439026 392 external_mini_cluster.cc:1428] Setting key 730d18cde014ac5763d2b1b163e73145
I20260929 19:04:58.439779 22338 raft_consensus.cc:348] T 00000000000000000000000000000000 P a228a78d489a42dd9cb3336b883ac964 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a228a78d489a42dd9cb3336b883ac964" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38185 } }
I20260929 19:04:58.439960 22338 raft_consensus.cc:374] T 00000000000000000000000000000000 P a228a78d489a42dd9cb3336b883ac964 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:04:58.439994 22338 raft_consensus.cc:729] T 00000000000000000000000000000000 P a228a78d489a42dd9cb3336b883ac964 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a228a78d489a42dd9cb3336b883ac964, State: Initialized, Role: FOLLOWER
I20260929 19:04:58.440128 22338 consensus_queue.cc:260] T 00000000000000000000000000000000 P a228a78d489a42dd9cb3336b883ac964 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a228a78d489a42dd9cb3336b883ac964" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38185 } }
I20260929 19:04:58.440215 22338 raft_consensus.cc:388] T 00000000000000000000000000000000 P a228a78d489a42dd9cb3336b883ac964 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 19:04:58.440301 22338 raft_consensus.cc:482] T 00000000000000000000000000000000 P a228a78d489a42dd9cb3336b883ac964 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 19:04:58.440366 22338 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a228a78d489a42dd9cb3336b883ac964 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:04:58.441053 22338 raft_consensus.cc:504] T 00000000000000000000000000000000 P a228a78d489a42dd9cb3336b883ac964 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a228a78d489a42dd9cb3336b883ac964" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38185 } }
I20260929 19:04:58.441160 22338 leader_election.cc:302] T 00000000000000000000000000000000 P a228a78d489a42dd9cb3336b883ac964 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a228a78d489a42dd9cb3336b883ac964; no voters:
I20260929 19:04:58.441324 22338 leader_election.cc:288] T 00000000000000000000000000000000 P a228a78d489a42dd9cb3336b883ac964 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 19:04:58.441382 22343 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a228a78d489a42dd9cb3336b883ac964 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:04:58.441560 22343 raft_consensus.cc:686] T 00000000000000000000000000000000 P a228a78d489a42dd9cb3336b883ac964 [term 1 LEADER]: Becoming Leader. State: Replica: a228a78d489a42dd9cb3336b883ac964, State: Running, Role: LEADER
I20260929 19:04:58.441579 22338 sys_catalog.cc:564] T 00000000000000000000000000000000 P a228a78d489a42dd9cb3336b883ac964 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 19:04:58.441663 22343 consensus_queue.cc:237] T 00000000000000000000000000000000 P a228a78d489a42dd9cb3336b883ac964 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a228a78d489a42dd9cb3336b883ac964" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38185 } }
I20260929 19:04:58.442058 22344 sys_catalog.cc:455] T 00000000000000000000000000000000 P a228a78d489a42dd9cb3336b883ac964 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a228a78d489a42dd9cb3336b883ac964" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a228a78d489a42dd9cb3336b883ac964" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38185 } } }
I20260929 19:04:58.442050 22343 sys_catalog.cc:455] T 00000000000000000000000000000000 P a228a78d489a42dd9cb3336b883ac964 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a228a78d489a42dd9cb3336b883ac964. Latest consensus state: current_term: 1 leader_uuid: "a228a78d489a42dd9cb3336b883ac964" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a228a78d489a42dd9cb3336b883ac964" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38185 } } }
I20260929 19:04:58.442195 22344 sys_catalog.cc:458] T 00000000000000000000000000000000 P a228a78d489a42dd9cb3336b883ac964 [sys.catalog]: This master's current role is: LEADER
I20260929 19:04:58.442205 22343 sys_catalog.cc:458] T 00000000000000000000000000000000 P a228a78d489a42dd9cb3336b883ac964 [sys.catalog]: This master's current role is: LEADER
I20260929 19:04:58.442436 22352 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 19:04:58.442947 22352 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 19:04:58.444873 22352 catalog_manager.cc:1347] Generated new cluster ID: 59d79c2e880349a0a58b6be89d076630
I20260929 19:04:58.444924 22352 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 29 19:04:58 dist-test-slave-lqkv krb5kdc[22257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708698, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 19:04:58.450285 22342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:58.439530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59333 (local address 127.0.98.62:38185)
0929 19:04:58.439764 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0929 19:04:58.439774 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 19:04:58.439809 (+ 35us) server_negotiation.cc:407] Connection header received
0929 19:04:58.439923 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:04:58.439930 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:04:58.439997 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:04:58.440149 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:04:58.440922 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:58.442417 (+ 1495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:58.443499 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:58.444024 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:58.444127 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:04:58.446605 (+ 2478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:04:58.446614 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:04:58.446614 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:04:58.446624 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:04:58.448294 (+ 1670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:04:58.448703 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:04:58.448707 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:04:58.448707 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:04:58.448856 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:04:58.449067 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:04:58.449068 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:04:58.449068 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:04:58.449313 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:04:58.449968 (+ 655us) server_negotiation.cc:1035] Waiting for connection context
0929 19:04:58.450033 (+ 65us) server_negotiation.cc:299] Negotiation successful
0929 19:04:58.450140 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":117,"threads_started":1}
I20260929 19:04:58.451174 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 19:04:58.462815 22352 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 19:04:58.463495 22342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:58.452512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57605 (local address 127.0.98.62:38185)
0929 19:04:58.452551 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0929 19:04:58.452553 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:04:58.457009 (+ 4456us) server_negotiation.cc:407] Connection header received
0929 19:04:58.457080 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:04:58.457081 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:04:58.457093 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:04:58.457159 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:04:58.457698 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:58.458187 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:58.458889 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:58.459112 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:58.459185 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:04:58.460407 (+ 1222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:04:58.460409 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:04:58.460409 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:04:58.460416 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:04:58.461831 (+ 1415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:04:58.462200 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:04:58.462201 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:04:58.462202 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:04:58.462258 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:04:58.462670 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:04:58.462671 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:04:58.462671 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:04:58.462808 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:04:58.462900 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0929 19:04:58.463319 (+ 419us) server_negotiation.cc:299] Negotiation successful
0929 19:04:58.463393 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260929 19:04:58.463717 22352 catalog_manager.cc:1504] Loading token signing keys...
I20260929 19:04:58.475268 22352 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a228a78d489a42dd9cb3336b883ac964: Generated new TSK 0
I20260929 19:04:58.475517 22352 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 19:04:58.482348 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUbnLHE/build/release/bin/kudu with pid 22270
2026-09-29T19:04:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1329 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790708447806542-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 19:04:58 dist-test-slave-lqkv krb5kdc[22369](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 19:04:58 dist-test-slave-lqkv krb5kdc[22369](info): set up 2 sockets
Sep 29 19:04:58 dist-test-slave-lqkv krb5kdc[22369](info): commencing operation
krb5kdc: starting...
W20260929 19:04:59.508975 392 mini_kdc.cc:121] Time spent starting KDC: real 1.009s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790708447806542-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790708447806542-392-0/krb5kdc/test-user.keytab.
Sep 29 19:04:59 dist-test-slave-lqkv krb5kdc[22369](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708699, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T19:04:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T19:04:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790708447806542-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790708447806542-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 19:04:59.627602 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUbnLHE/build/release/bin/kudu
/tmp/dist-test-taskUbnLHE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790708447806542-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790708447806542-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790708447806542-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790708447806542-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43959
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790708447806542-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43959
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790708447806542-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790708447806542-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790708447806542-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:04:59.712160 22382 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:04:59.712410 22382 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:04:59.712451 22382 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:04:59.713703 22382 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 19:04:59.713771 22382 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:04:59.713795 22382 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:04:59.713828 22382 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 19:04:59.713856 22382 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 19:04:59.715368 22382 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790708447806542-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790708447806542-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43959
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790708447806542-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43959
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790708447806542-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22382
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790708447806542-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 18:09:13 UTC on ccycloud.quasar-hpkalf.root.comops.site
build id 4680913
I20260929 19:04:59.715643 22382 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:04:59.715863 22382 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:04:59.720757 22388 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:04:59.720757 22387 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:04:59.720827 22390 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:04:59.720959 22382 server_base.cc:1034] running on GCE node
I20260929 19:04:59.721194 22382 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:04:59.721526 22382 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:04:59.722683 22382 hybrid_clock.cc:648] HybridClock initialized: now 1790708699722664 us; error 47 us; skew 500 ppm
Sep 29 19:04:59 dist-test-slave-lqkv krb5kdc[22369](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708699, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:04:59.725077 22382 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 19:04:59.725497 22382 webserver.cc:460] Webserver started at http://127.0.98.62:41119/ using document root <none> and password file <none>
I20260929 19:04:59.725723 22382 fs_manager.cc:359] Metadata directory not provided
I20260929 19:04:59.725786 22382 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:04:59.725927 22382 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:04:59.726955 22382 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790708447806542-392-0/minicluster-data/master-0/data/instance:
uuid: "65720dfcf79e4e84ab64fe98d62863b7"
format_stamp: "Formatted at 2026-09-29 19:04:59 on dist-test-slave-lqkv"
server_key: "1262ff001b244ff3a5d98427ced1d034"
server_key_iv: "b118a3925fbb6078a147746d30cfa639"
server_key_version: "encryptionkey@0"
I20260929 19:04:59.727304 22382 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790708447806542-392-0/minicluster-data/master-0/wal/instance:
uuid: "65720dfcf79e4e84ab64fe98d62863b7"
format_stamp: "Formatted at 2026-09-29 19:04:59 on dist-test-slave-lqkv"
server_key: "1262ff001b244ff3a5d98427ced1d034"
server_key_iv: "b118a3925fbb6078a147746d30cfa639"
server_key_version: "encryptionkey@0"
I20260929 19:04:59.728672 22382 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260929 19:04:59.729458 22397 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:04:59.729624 22382 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 19:04:59.729718 22382 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790708447806542-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790708447806542-392-0/minicluster-data/master-0/wal
uuid: "65720dfcf79e4e84ab64fe98d62863b7"
format_stamp: "Formatted at 2026-09-29 19:04:59 on dist-test-slave-lqkv"
server_key: "1262ff001b244ff3a5d98427ced1d034"
server_key_iv: "b118a3925fbb6078a147746d30cfa639"
server_key_version: "encryptionkey@0"
I20260929 19:04:59.729794 22382 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790708447806542-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790708447806542-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790708447806542-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:04:59.747236 22382 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:04:59.749136 22382 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:04:59.749290 22382 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:04:59.753145 22382 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43959
I20260929 19:04:59.753182 22449 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43959 every 8 connection(s)
I20260929 19:04:59.753535 22382 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790708447806542-392-0/minicluster-data/master-0/data/info.pb
I20260929 19:04:59.754380 22450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:04:59.756628 22450 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65720dfcf79e4e84ab64fe98d62863b7: Bootstrap starting.
I20260929 19:04:59.757273 22450 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 65720dfcf79e4e84ab64fe98d62863b7: Neither blocks nor log segments found. Creating new log.
I20260929 19:04:59.757555 22450 log.cc:824] T 00000000000000000000000000000000 P 65720dfcf79e4e84ab64fe98d62863b7: Log is configured to *not* fsync() on all Append() calls
I20260929 19:04:59.758224 22450 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65720dfcf79e4e84ab64fe98d62863b7: No bootstrap required, opened a new log
I20260929 19:04:59.759759 22450 raft_consensus.cc:348] T 00000000000000000000000000000000 P 65720dfcf79e4e84ab64fe98d62863b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65720dfcf79e4e84ab64fe98d62863b7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43959 } }
I20260929 19:04:59.759892 22450 raft_consensus.cc:374] T 00000000000000000000000000000000 P 65720dfcf79e4e84ab64fe98d62863b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:04:59.759945 22450 raft_consensus.cc:729] T 00000000000000000000000000000000 P 65720dfcf79e4e84ab64fe98d62863b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65720dfcf79e4e84ab64fe98d62863b7, State: Initialized, Role: FOLLOWER
I20260929 19:04:59.760114 22450 consensus_queue.cc:260] T 00000000000000000000000000000000 P 65720dfcf79e4e84ab64fe98d62863b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65720dfcf79e4e84ab64fe98d62863b7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43959 } }
I20260929 19:04:59.760196 22450 raft_consensus.cc:388] T 00000000000000000000000000000000 P 65720dfcf79e4e84ab64fe98d62863b7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 19:04:59.760244 22450 raft_consensus.cc:482] T 00000000000000000000000000000000 P 65720dfcf79e4e84ab64fe98d62863b7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 19:04:59.760294 22450 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 65720dfcf79e4e84ab64fe98d62863b7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:04:59.760848 22450 raft_consensus.cc:504] T 00000000000000000000000000000000 P 65720dfcf79e4e84ab64fe98d62863b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65720dfcf79e4e84ab64fe98d62863b7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43959 } }
I20260929 19:04:59.760957 22450 leader_election.cc:302] T 00000000000000000000000000000000 P 65720dfcf79e4e84ab64fe98d62863b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 65720dfcf79e4e84ab64fe98d62863b7; no voters:
I20260929 19:04:59.761132 22450 leader_election.cc:288] T 00000000000000000000000000000000 P 65720dfcf79e4e84ab64fe98d62863b7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 19:04:59.761171 22453 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 65720dfcf79e4e84ab64fe98d62863b7 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:04:59.761338 22453 raft_consensus.cc:686] T 00000000000000000000000000000000 P 65720dfcf79e4e84ab64fe98d62863b7 [term 1 LEADER]: Becoming Leader. State: Replica: 65720dfcf79e4e84ab64fe98d62863b7, State: Running, Role: LEADER
I20260929 19:04:59.761369 22450 sys_catalog.cc:564] T 00000000000000000000000000000000 P 65720dfcf79e4e84ab64fe98d62863b7 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 19:04:59.761447 22453 consensus_queue.cc:237] T 00000000000000000000000000000000 P 65720dfcf79e4e84ab64fe98d62863b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65720dfcf79e4e84ab64fe98d62863b7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43959 } }
I20260929 19:04:59.761813 22455 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65720dfcf79e4e84ab64fe98d62863b7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 65720dfcf79e4e84ab64fe98d62863b7. Latest consensus state: current_term: 1 leader_uuid: "65720dfcf79e4e84ab64fe98d62863b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65720dfcf79e4e84ab64fe98d62863b7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43959 } } }
I20260929 19:04:59.761809 22454 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65720dfcf79e4e84ab64fe98d62863b7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "65720dfcf79e4e84ab64fe98d62863b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65720dfcf79e4e84ab64fe98d62863b7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43959 } } }
I20260929 19:04:59.761929 22455 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65720dfcf79e4e84ab64fe98d62863b7 [sys.catalog]: This master's current role is: LEADER
I20260929 19:04:59.762002 22454 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65720dfcf79e4e84ab64fe98d62863b7 [sys.catalog]: This master's current role is: LEADER
I20260929 19:04:59.762408 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUbnLHE/build/release/bin/kudu as pid 22382
I20260929 19:04:59.762522 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790708447806542-392-0/minicluster-data/master-0/wal/instance
I20260929 19:04:59.762691 392 external_mini_cluster.cc:1428] Setting key 3848d52a310e65d98ff3ae0de4fbfa1e
I20260929 19:04:59.763041 22457 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 19:04:59.763566 22457 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 19:04:59.765101 22457 catalog_manager.cc:1347] Generated new cluster ID: f2c493bcc2af442897160c4448578c0e
I20260929 19:04:59.765153 22457 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 19:04:59.768261 22467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:59.763357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34281 (local address 127.0.98.62:43959)
0929 19:04:59.763515 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0929 19:04:59.763525 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 19:04:59.763553 (+ 28us) server_negotiation.cc:407] Connection header received
0929 19:04:59.763669 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:04:59.763676 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:04:59.763746 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:04:59.763877 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:04:59.764074 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:59.765600 (+ 1526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:59.766355 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:59.766932 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:59.767035 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:04:59.767643 (+ 608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:04:59.767647 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:04:59.767647 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 19:04:59.767840 (+ 193us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:04:59.767874 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 19:04:59.767887 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:04:59.767941 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0929 19:04:59.768009 (+ 68us) server_negotiation.cc:299] Negotiation successful
0929 19:04:59.768067 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":71,"threads_started":1}
I20260929 19:04:59.769142 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 19:04:59.773347 22467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:04:59.770502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41139 (local address 127.0.98.62:43959)
0929 19:04:59.770538 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0929 19:04:59.770540 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:04:59.770641 (+ 101us) server_negotiation.cc:407] Connection header received
0929 19:04:59.770683 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:04:59.770684 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:04:59.770692 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:04:59.770752 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:04:59.770842 (+ 90us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:59.771540 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:59.772179 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:04:59.772460 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:04:59.772517 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:04:59.773122 (+ 605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:04:59.773123 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:04:59.773124 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 19:04:59.773130 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:04:59.773143 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 19:04:59.773147 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:04:59.773200 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0929 19:04:59.773258 (+ 58us) server_negotiation.cc:299] Negotiation successful
0929 19:04:59.773279 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260929 19:04:59.793757 22457 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 19:04:59.794282 22457 catalog_manager.cc:1504] Loading token signing keys...
I20260929 19:04:59.803151 22457 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 65720dfcf79e4e84ab64fe98d62863b7: Generated new TSK 0
I20260929 19:04:59.803351 22457 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 19:04:59.808010 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUbnLHE/build/release/bin/kudu with pid 22382
2026-09-29T19:04:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1325 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790708447806542-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 19:04:59 dist-test-slave-lqkv krb5kdc[22481](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 19:04:59 dist-test-slave-lqkv krb5kdc[22481](info): set up 2 sockets
Sep 29 19:04:59 dist-test-slave-lqkv krb5kdc[22481](info): commencing operation
krb5kdc: starting...
W20260929 19:05:00.837062 392 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790708447806542-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790708447806542-392-0/krb5kdc/test-user.keytab.
Sep 29 19:05:00 dist-test-slave-lqkv krb5kdc[22481](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708700, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T19:05:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T19:05:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790708447806542-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790708447806542-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 19:05:00.955703 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUbnLHE/build/release/bin/kudu
/tmp/dist-test-taskUbnLHE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790708447806542-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790708447806542-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790708447806542-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790708447806542-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43907
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790708447806542-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43907
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790708447806542-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790708447806542-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790708447806542-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:05:01.036674 22494 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:05:01.036959 22494 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:05:01.037003 22494 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:05:01.038184 22494 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 19:05:01.038254 22494 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:05:01.038280 22494 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:05:01.038324 22494 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 19:05:01.038350 22494 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 19:05:01.039844 22494 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790708447806542-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790708447806542-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43907
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790708447806542-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43907
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790708447806542-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22494
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790708447806542-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 18:09:13 UTC on ccycloud.quasar-hpkalf.root.comops.site
build id 4680913
I20260929 19:05:01.040119 22494 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:05:01.040337 22494 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:05:01.045380 22500 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:05:01.045375 22499 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:05:01.045374 22502 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:05:01.045485 22494 server_base.cc:1034] running on GCE node
I20260929 19:05:01.045801 22494 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:05:01.046095 22494 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:05:01.047250 22494 hybrid_clock.cc:648] HybridClock initialized: now 1790708701047237 us; error 31 us; skew 500 ppm
Sep 29 19:05:01 dist-test-slave-lqkv krb5kdc[22481](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708701, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:05:01.049955 22494 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 19:05:01.050345 22494 webserver.cc:460] Webserver started at http://127.0.98.62:39559/ using document root <none> and password file <none>
I20260929 19:05:01.050540 22494 fs_manager.cc:359] Metadata directory not provided
I20260929 19:05:01.050581 22494 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:05:01.050670 22494 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:05:01.051728 22494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790708447806542-392-0/minicluster-data/master-0/data/instance:
uuid: "5970904b244448248133093a72394a85"
format_stamp: "Formatted at 2026-09-29 19:05:01 on dist-test-slave-lqkv"
server_key: "5234c39790bdbbf3975ea4143c39630c"
server_key_iv: "9b07e3728203874034499635f7f54cd9"
server_key_version: "encryptionkey@0"
I20260929 19:05:01.052028 22494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790708447806542-392-0/minicluster-data/master-0/wal/instance:
uuid: "5970904b244448248133093a72394a85"
format_stamp: "Formatted at 2026-09-29 19:05:01 on dist-test-slave-lqkv"
server_key: "5234c39790bdbbf3975ea4143c39630c"
server_key_iv: "9b07e3728203874034499635f7f54cd9"
server_key_version: "encryptionkey@0"
I20260929 19:05:01.053356 22494 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 19:05:01.054075 22509 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:05:01.054235 22494 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 19:05:01.054334 22494 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790708447806542-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790708447806542-392-0/minicluster-data/master-0/wal
uuid: "5970904b244448248133093a72394a85"
format_stamp: "Formatted at 2026-09-29 19:05:01 on dist-test-slave-lqkv"
server_key: "5234c39790bdbbf3975ea4143c39630c"
server_key_iv: "9b07e3728203874034499635f7f54cd9"
server_key_version: "encryptionkey@0"
I20260929 19:05:01.054433 22494 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790708447806542-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790708447806542-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790708447806542-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:05:01.057729 22494 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:05:01.060015 22494 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:05:01.060164 22494 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:05:01.063911 22494 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43907
I20260929 19:05:01.063979 22561 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43907 every 8 connection(s)
I20260929 19:05:01.064296 22494 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790708447806542-392-0/minicluster-data/master-0/data/info.pb
I20260929 19:05:01.065335 22562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:05:01.067626 22562 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5970904b244448248133093a72394a85: Bootstrap starting.
I20260929 19:05:01.068203 22562 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5970904b244448248133093a72394a85: Neither blocks nor log segments found. Creating new log.
I20260929 19:05:01.068511 22562 log.cc:824] T 00000000000000000000000000000000 P 5970904b244448248133093a72394a85: Log is configured to *not* fsync() on all Append() calls
I20260929 19:05:01.069408 22562 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5970904b244448248133093a72394a85: No bootstrap required, opened a new log
I20260929 19:05:01.070261 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUbnLHE/build/release/bin/kudu as pid 22494
I20260929 19:05:01.070412 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790708447806542-392-0/minicluster-data/master-0/wal/instance
I20260929 19:05:01.070612 392 external_mini_cluster.cc:1428] Setting key 781ee9bdba9791d9bd748e3e16134926
I20260929 19:05:01.071419 22562 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5970904b244448248133093a72394a85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5970904b244448248133093a72394a85" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43907 } }
I20260929 19:05:01.071579 22562 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5970904b244448248133093a72394a85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 19:05:01.071642 22562 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5970904b244448248133093a72394a85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5970904b244448248133093a72394a85, State: Initialized, Role: FOLLOWER
I20260929 19:05:01.071794 22562 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5970904b244448248133093a72394a85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5970904b244448248133093a72394a85" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43907 } }
I20260929 19:05:01.071903 22562 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5970904b244448248133093a72394a85 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 19:05:01.071938 22562 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5970904b244448248133093a72394a85 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 19:05:01.071987 22562 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5970904b244448248133093a72394a85 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:05:01.072852 22566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:01.071143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48093 (local address 127.0.98.62:43907)
0929 19:05:01.071331 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0929 19:05:01.071339 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 19:05:01.071364 (+ 25us) server_negotiation.cc:407] Connection header received
0929 19:05:01.071458 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:05:01.071463 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:05:01.071488 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:05:01.071639 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:05:01.072337 (+ 698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:05:01.072341 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:05:01.072341 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 19:05:01.072345 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:05:01.072393 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 19:05:01.072411 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:05:01.072497 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 19:05:01.072550 (+ 53us) server_negotiation.cc:299] Negotiation successful
0929 19:05:01.072600 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":100,"threads_started":1}
I20260929 19:05:01.073187 22562 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5970904b244448248133093a72394a85 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5970904b244448248133093a72394a85" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43907 } }
I20260929 19:05:01.073292 22562 leader_election.cc:302] T 00000000000000000000000000000000 P 5970904b244448248133093a72394a85 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5970904b244448248133093a72394a85; no voters:
I20260929 19:05:01.073470 22562 leader_election.cc:288] T 00000000000000000000000000000000 P 5970904b244448248133093a72394a85 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 19:05:01.073490 22567 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5970904b244448248133093a72394a85 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:05:01.073596 22567 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5970904b244448248133093a72394a85 [term 1 LEADER]: Becoming Leader. State: Replica: 5970904b244448248133093a72394a85, State: Running, Role: LEADER
I20260929 19:05:01.073721 22562 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5970904b244448248133093a72394a85 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 19:05:01.073704 22567 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5970904b244448248133093a72394a85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5970904b244448248133093a72394a85" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43907 } }
I20260929 19:05:01.073807 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 19:05:01.074072 22567 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5970904b244448248133093a72394a85 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5970904b244448248133093a72394a85. Latest consensus state: current_term: 1 leader_uuid: "5970904b244448248133093a72394a85" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5970904b244448248133093a72394a85" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43907 } } }
I20260929 19:05:01.074139 22567 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5970904b244448248133093a72394a85 [sys.catalog]: This master's current role is: LEADER
I20260929 19:05:01.074375 22569 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5970904b244448248133093a72394a85 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5970904b244448248133093a72394a85" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5970904b244448248133093a72394a85" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43907 } } }
I20260929 19:05:01.074440 22569 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5970904b244448248133093a72394a85 [sys.catalog]: This master's current role is: LEADER
I20260929 19:05:01.074851 22572 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 19:05:01.075389 22572 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 19:05:01.077082 22572 catalog_manager.cc:1347] Generated new cluster ID: bd06f762bcbf47eb85d8a2a4ffc1be7e
I20260929 19:05:01.077133 22572 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 19:05:01.083421 22572 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 19:05:01.083633 22572 catalog_manager.cc:1504] Loading token signing keys...
I20260929 19:05:01.084414 22566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:01.075601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42419 (local address 127.0.98.62:43907)
0929 19:05:01.075712 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0929 19:05:01.075715 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 19:05:01.083318 (+ 7603us) server_negotiation.cc:407] Connection header received
0929 19:05:01.083379 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:05:01.083380 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:05:01.083386 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:05:01.083463 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:05:01.084176 (+ 713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:05:01.084177 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:05:01.084178 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 19:05:01.084182 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:05:01.084197 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 19:05:01.084202 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:05:01.084251 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0929 19:05:01.084312 (+ 61us) server_negotiation.cc:299] Negotiation successful
0929 19:05:01.084328 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260929 19:05:01.093291 22572 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5970904b244448248133093a72394a85: Generated new TSK 0
I20260929 19:05:01.093534 22572 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 19:05:01.101253 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUbnLHE/build/release/bin/kudu with pid 22494
2026-09-29T19:05:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1293 ms)
[----------] 18 tests from AuthTokenIssuingTest (22593 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708447806542-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 19:05:01 dist-test-slave-lqkv krb5kdc[22593](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 19:05:01 dist-test-slave-lqkv krb5kdc[22593](info): set up 2 sockets
Sep 29 19:05:01 dist-test-slave-lqkv krb5kdc[22593](info): commencing operation
krb5kdc: starting...
W20260929 19:05:02.165033 392 mini_kdc.cc:121] Time spent starting KDC: real 1.046s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708447806542-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708447806542-392-0/krb5kdc/test-user.keytab.
Sep 29 19:05:02 dist-test-slave-lqkv krb5kdc[22593](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708702, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T19:05:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T19:05:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708447806542-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708447806542-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 19:05:02.280179 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUbnLHE/build/release/bin/kudu
/tmp/dist-test-taskUbnLHE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708447806542-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708447806542-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708447806542-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708447806542-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36031
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708447806542-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36031,127.0.98.61:43091,127.0.98.60:37183
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708447806542-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708447806542-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708447806542-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:05:02.360435 22606 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 19:05:02.360760 22606 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:05:02.360792 22606 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:05:02.360867 22606 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:05:02.362076 22606 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 19:05:02.362126 22606 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:05:02.362140 22606 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:05:02.362154 22606 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 19:05:02.362165 22606 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 19:05:02.363622 22606 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708447806542-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708447806542-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36031,127.0.98.61:43091,127.0.98.60:37183
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708447806542-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36031
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708447806542-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22606
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708447806542-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 18:09:13 UTC on ccycloud.quasar-hpkalf.root.comops.site
build id 4680913
I20260929 19:05:02.363830 22606 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:05:02.364001 22606 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:05:02.368894 22611 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:05:02.368902 22612 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:05:02.368937 22614 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:05:02.369089 22606 server_base.cc:1034] running on GCE node
I20260929 19:05:02.369459 22606 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:05:02.369747 22606 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:05:02.370925 22606 hybrid_clock.cc:648] HybridClock initialized: now 1790708702370917 us; error 34 us; skew 500 ppm
Sep 29 19:05:02 dist-test-slave-lqkv krb5kdc[22593](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708702, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:05:02.376174 22606 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 19:05:02.376580 22606 webserver.cc:460] Webserver started at http://127.0.98.62:36269/ using document root <none> and password file <none>
I20260929 19:05:02.376768 22606 fs_manager.cc:359] Metadata directory not provided
I20260929 19:05:02.376868 22606 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:05:02.377025 22606 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:05:02.378106 22606 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708447806542-392-0/minicluster-data/master-0/data/instance:
uuid: "315bce7e99d447a78fc04e183250a8a6"
format_stamp: "Formatted at 2026-09-29 19:05:02 on dist-test-slave-lqkv"
server_key: "ccf9aff7e6fcc2bd4eca0c4bd622cb99"
server_key_iv: "3794f01d94bb8b0081f0b6f2785c73ce"
server_key_version: "encryptionkey@0"
I20260929 19:05:02.378499 22606 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708447806542-392-0/minicluster-data/master-0/wal/instance:
uuid: "315bce7e99d447a78fc04e183250a8a6"
format_stamp: "Formatted at 2026-09-29 19:05:02 on dist-test-slave-lqkv"
server_key: "ccf9aff7e6fcc2bd4eca0c4bd622cb99"
server_key_iv: "3794f01d94bb8b0081f0b6f2785c73ce"
server_key_version: "encryptionkey@0"
I20260929 19:05:02.379931 22606 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 19:05:02.380707 22621 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:05:02.380959 22606 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 19:05:02.381053 22606 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708447806542-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708447806542-392-0/minicluster-data/master-0/wal
uuid: "315bce7e99d447a78fc04e183250a8a6"
format_stamp: "Formatted at 2026-09-29 19:05:02 on dist-test-slave-lqkv"
server_key: "ccf9aff7e6fcc2bd4eca0c4bd622cb99"
server_key_iv: "3794f01d94bb8b0081f0b6f2785c73ce"
server_key_version: "encryptionkey@0"
I20260929 19:05:02.381103 22606 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708447806542-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708447806542-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708447806542-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:05:02.401960 22606 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:05:02.404320 22606 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:05:02.404448 22606 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:05:02.408386 22606 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36031
I20260929 19:05:02.408459 22673 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36031 every 8 connection(s)
I20260929 19:05:02.408795 22606 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708447806542-392-0/minicluster-data/master-0/data/info.pb
I20260929 19:05:02.409843 22674 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:05:02.411389 22674 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36031 } has no permanent_uuid. Determining permanent_uuid...
I20260929 19:05:02.414758 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUbnLHE/build/release/bin/kudu as pid 22606
I20260929 19:05:02.414868 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708447806542-392-0/minicluster-data/master-0/wal/instance
I20260929 19:05:02.415051 392 external_mini_cluster.cc:1428] Setting key e6d385ddccd6e89764e02661fc08e1b3
Sep 29 19:05:02 dist-test-slave-lqkv krb5kdc[22593](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708702, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 19:05:02.421361 22677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:02.412225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60726 (local address 127.0.98.62:36031)
0929 19:05:02.412338 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0929 19:05:02.412342 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 19:05:02.412372 (+ 30us) server_negotiation.cc:407] Connection header received
0929 19:05:02.412877 (+ 505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:05:02.412879 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:05:02.412896 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:05:02.412974 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:05:02.413590 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:02.414395 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:02.415362 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:02.415875 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:02.415985 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:05:02.418231 (+ 2246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:05:02.418238 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:05:02.418239 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:05:02.418245 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:05:02.419839 (+ 1594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:05:02.420217 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:05:02.420219 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:05:02.420220 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:05:02.420295 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:05:02.420518 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:05:02.420519 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:05:02.420519 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:05:02.420711 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:05:02.420797 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 19:05:02.420993 (+ 196us) server_negotiation.cc:299] Negotiation successful
0929 19:05:02.421126 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81,"thread_start_us":57,"threads_started":1}
I20260929 19:05:02.421363 22676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:02.412078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36031 (local address 127.0.0.1:60726)
0929 19:05:02.412233 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0929 19:05:02.412272 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 19:05:02.412706 (+ 434us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:05:02.413010 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:05:02.413013 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:05:02.413345 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:05:02.413505 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:05:02.413512 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:05:02.414472 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:05:02.414473 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:05:02.415292 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:05:02.415294 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:05:02.415970 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:05:02.415971 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:05:02.416034 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:05:02.416580 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:05:02.416592 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:05:02.418147 (+ 1555us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:05:02.419928 (+ 1781us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:05:02.419935 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:05:02.419936 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:05:02.420128 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:05:02.420357 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:05:02.420358 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:05:02.420359 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:05:02.420441 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:05:02.420776 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:05:02.420781 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:05:02.420870 (+ 89us) client_negotiation.cc:769] Sending connection context
0929 19:05:02.420970 (+ 100us) client_negotiation.cc:240] Negotiation successful
0929 19:05:02.421187 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"mutex_wait_us":80,"thread_start_us":66,"threads_started":1}
I20260929 19:05:02.422034 22674 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43091 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
I20260929 19:05:02.422359 22676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:02.422250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43091 (local address 127.0.0.1:52120)
0929 19:05:02.422292 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0929 19:05:02.422336 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260929 19:05:02.422484 22625 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43091: connect: Connection refused (error 111)
I20260929 19:05:02.422855 22676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:02.422776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43091 (local address 127.0.0.1:52126)
0929 19:05:02.422816 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0929 19:05:02.422840 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260929 19:05:02.422948 22674 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:43091: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43091: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708447806542-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708447806542-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260929 19:05:02.439225 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUbnLHE/build/release/bin/kudu
/tmp/dist-test-taskUbnLHE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708447806542-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708447806542-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708447806542-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708447806542-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:43091
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708447806542-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36031,127.0.98.61:43091,127.0.98.60:37183
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708447806542-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708447806542-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708447806542-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 19:05:02.472163 22674 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43091 } attempt: 1
I20260929 19:05:02.472678 22676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:02.472531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43091 (local address 127.0.0.1:52134)
0929 19:05:02.472591 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0929 19:05:02.472632 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260929 19:05:02.473099 22676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:02.473011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43091 (local address 127.0.0.1:52140)
0929 19:05:02.473054 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0929 19:05:02.473080 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260929 19:05:02.473222 22674 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:43091: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43091: connect: Connection refused (error 111)
W20260929 19:05:02.519794 22681 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 19:05:02.520011 22681 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:05:02.520030 22681 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:05:02.520048 22681 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:05:02.521225 22681 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 19:05:02.521265 22681 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:05:02.521279 22681 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:05:02.521317 22681 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 19:05:02.521342 22681 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 19:05:02.522833 22681 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708447806542-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708447806542-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36031,127.0.98.61:43091,127.0.98.60:37183
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708447806542-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:43091
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708447806542-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22681
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708447806542-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 18:09:13 UTC on ccycloud.quasar-hpkalf.root.comops.site
build id 4680913
I20260929 19:05:02.523083 22681 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:05:02.523296 22681 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:05:02.528110 22689 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:05:02.528098 22687 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:05:02.528100 22686 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:05:02.528188 22681 server_base.cc:1034] running on GCE node
I20260929 19:05:02.528537 22681 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:05:02.528863 22681 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:05:02.530023 22681 hybrid_clock.cc:648] HybridClock initialized: now 1790708702530015 us; error 28 us; skew 500 ppm
Sep 29 19:05:02 dist-test-slave-lqkv krb5kdc[22593](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708702, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:05:02.532469 22681 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260929 19:05:02.532908 22681 webserver.cc:460] Webserver started at http://127.0.98.61:37705/ using document root <none> and password file <none>
I20260929 19:05:02.533104 22681 fs_manager.cc:359] Metadata directory not provided
I20260929 19:05:02.533144 22681 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:05:02.533234 22681 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:05:02.534237 22681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708447806542-392-0/minicluster-data/master-1/data/instance:
uuid: "d033f97a05654c89a2c695a4d3f638a7"
format_stamp: "Formatted at 2026-09-29 19:05:02 on dist-test-slave-lqkv"
server_key: "e149ae72b22bb2aee418486d9d9bb78d"
server_key_iv: "890918846e28e804672c5e1c10c78680"
server_key_version: "encryptionkey@0"
I20260929 19:05:02.534528 22681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708447806542-392-0/minicluster-data/master-1/wal/instance:
uuid: "d033f97a05654c89a2c695a4d3f638a7"
format_stamp: "Formatted at 2026-09-29 19:05:02 on dist-test-slave-lqkv"
server_key: "e149ae72b22bb2aee418486d9d9bb78d"
server_key_iv: "890918846e28e804672c5e1c10c78680"
server_key_version: "encryptionkey@0"
I20260929 19:05:02.535714 22681 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260929 19:05:02.536379 22696 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:05:02.536540 22681 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 19:05:02.536619 22681 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708447806542-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708447806542-392-0/minicluster-data/master-1/wal
uuid: "d033f97a05654c89a2c695a4d3f638a7"
format_stamp: "Formatted at 2026-09-29 19:05:02 on dist-test-slave-lqkv"
server_key: "e149ae72b22bb2aee418486d9d9bb78d"
server_key_iv: "890918846e28e804672c5e1c10c78680"
server_key_version: "encryptionkey@0"
I20260929 19:05:02.536674 22681 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708447806542-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708447806542-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708447806542-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:05:02.541369 22674 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43091 } attempt: 2
I20260929 19:05:02.541788 22676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:02.541666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43091 (local address 127.0.0.1:52144)
0929 19:05:02.541722 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0929 19:05:02.541759 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260929 19:05:02.542099 22676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:02.542004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43091 (local address 127.0.0.1:52152)
0929 19:05:02.542045 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0929 19:05:02.542070 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260929 19:05:02.542181 22674 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:43091: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43091: connect: Connection refused (error 111)
I20260929 19:05:02.549214 22681 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:05:02.551401 22681 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:05:02.551559 22681 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:05:02.555354 22681 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:43091
I20260929 19:05:02.555389 22748 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:43091 every 8 connection(s)
I20260929 19:05:02.555759 22681 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708447806542-392-0/minicluster-data/master-1/data/info.pb
I20260929 19:05:02.556715 22749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:05:02.558233 22749 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36031 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 19:05:02 dist-test-slave-lqkv krb5kdc[22593](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708702, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 19:05:02.564050 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUbnLHE/build/release/bin/kudu as pid 22681
I20260929 19:05:02.564168 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708447806542-392-0/minicluster-data/master-1/wal/instance
I20260929 19:05:02.564363 392 external_mini_cluster.cc:1428] Setting key cb63845898019884ce326247b7b19da7
I20260929 19:05:02.567130 22677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:02.558861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60738 (local address 127.0.98.62:36031)
0929 19:05:02.558914 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0929 19:05:02.558916 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:05:02.559100 (+ 184us) server_negotiation.cc:407] Connection header received
0929 19:05:02.559555 (+ 455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:05:02.559556 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:05:02.559568 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:05:02.559613 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:05:02.560202 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:02.560685 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:02.561692 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:02.561887 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:02.561960 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:05:02.564145 (+ 2185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:05:02.564147 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:05:02.564148 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:05:02.564156 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:05:02.565668 (+ 1512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:05:02.566094 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:05:02.566096 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:05:02.566096 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:05:02.566155 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:05:02.566409 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:05:02.566410 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:05:02.566411 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:05:02.566562 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:05:02.566626 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0929 19:05:02.566890 (+ 264us) server_negotiation.cc:299] Negotiation successful
0929 19:05:02.566984 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260929 19:05:02.567152 22751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:02.558811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36031 (local address 127.0.0.1:60738)
0929 19:05:02.558973 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0929 19:05:02.559021 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 19:05:02.559456 (+ 435us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:05:02.559689 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:05:02.559692 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:05:02.559932 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:05:02.560134 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:05:02.560138 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:05:02.560758 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:05:02.560758 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:05:02.561585 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:05:02.561587 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:05:02.561974 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:05:02.561975 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:05:02.562088 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:05:02.562586 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:05:02.562594 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:05:02.564030 (+ 1436us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:05:02.565772 (+ 1742us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:05:02.565779 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:05:02.565782 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:05:02.566010 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:05:02.566227 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:05:02.566228 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:05:02.566228 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:05:02.566338 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:05:02.566630 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:05:02.566636 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:05:02.566757 (+ 121us) client_negotiation.cc:769] Sending connection context
0929 19:05:02.566851 (+ 94us) client_negotiation.cc:240] Negotiation successful
0929 19:05:02.566984 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":81,"threads_started":1}
I20260929 19:05:02.567919 22749 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43091 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Sep 29 19:05:02 dist-test-slave-lqkv krb5kdc[22593](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708702, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260929 19:05:02.577425 22751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:02.568189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43091 (local address 127.0.0.1:52156)
0929 19:05:02.568245 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0929 19:05:02.568258 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 19:05:02.568305 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:05:02.568592 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:05:02.568593 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:05:02.568733 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:05:02.568776 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:05:02.568778 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:05:02.569664 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:05:02.569665 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:05:02.570164 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:05:02.570165 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:05:02.570640 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:05:02.570641 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:05:02.570704 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:05:02.571175 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:05:02.571179 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:05:02.574672 (+ 3493us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:05:02.576360 (+ 1688us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:05:02.576361 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:05:02.576361 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:05:02.576532 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:05:02.576747 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:05:02.576748 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:05:02.576748 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:05:02.576840 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:05:02.577108 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:05:02.577109 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:05:02.577144 (+ 35us) client_negotiation.cc:769] Sending connection context
0929 19:05:02.577201 (+ 57us) client_negotiation.cc:240] Negotiation successful
0929 19:05:02.577306 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260929 19:05:02.577603 22753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:02.568286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52156 (local address 127.0.98.61:43091)
0929 19:05:02.568464 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0929 19:05:02.568467 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 19:05:02.568482 (+ 15us) server_negotiation.cc:407] Connection header received
0929 19:05:02.568500 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:05:02.568501 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:05:02.568511 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:05:02.568592 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:05:02.568900 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:02.569572 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:02.570253 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:02.570558 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:02.570642 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:05:02.574761 (+ 4119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:05:02.574769 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:05:02.574769 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:05:02.574775 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:05:02.576288 (+ 1513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:05:02.576628 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:05:02.576632 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:05:02.576632 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:05:02.576685 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:05:02.576893 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:05:02.576894 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:05:02.576894 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:05:02.577041 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:05:02.577130 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0929 19:05:02.577239 (+ 109us) server_negotiation.cc:299] Negotiation successful
0929 19:05:02.577412 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":78,"server-negotiator.queue_time_us":155,"thread_start_us":92,"threads_started":1}
I20260929 19:05:02.578183 22749 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37183 } has no permanent_uuid. Determining permanent_uuid...
I20260929 19:05:02.578527 22751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:02.578414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37183 (local address 127.0.0.1:35172)
0929 19:05:02.578460 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0929 19:05:02.578502 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260929 19:05:02.578680 22700 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37183: connect: Connection refused (error 111)
I20260929 19:05:02.579113 22751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:02.579030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37183 (local address 127.0.0.1:35180)
0929 19:05:02.579074 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0929 19:05:02.579098 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260929 19:05:02.579223 22749 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:37183: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37183: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708447806542-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708447806542-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260929 19:05:02.587610 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUbnLHE/build/release/bin/kudu
/tmp/dist-test-taskUbnLHE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708447806542-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708447806542-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708447806542-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708447806542-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:37183
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708447806542-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36031,127.0.98.61:43091,127.0.98.60:37183
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708447806542-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708447806542-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708447806542-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 19:05:02.628463 22749 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37183 } attempt: 1
I20260929 19:05:02.629102 22751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:02.628918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37183 (local address 127.0.0.1:35196)
0929 19:05:02.629013 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0929 19:05:02.629060 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260929 19:05:02.629504 22751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:02.629420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37183 (local address 127.0.0.1:35206)
0929 19:05:02.629460 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0929 19:05:02.629487 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260929 19:05:02.629606 22749 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:37183: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37183: connect: Connection refused (error 111)
I20260929 19:05:02.633358 22674 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43091 } attempt: 3
Sep 29 19:05:02 dist-test-slave-lqkv krb5kdc[22593](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708702, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260929 19:05:02.640120 22676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:02.633680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43091 (local address 127.0.0.1:52164)
0929 19:05:02.633720 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0929 19:05:02.633731 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 19:05:02.633802 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:05:02.633950 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:05:02.633951 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:05:02.634122 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:05:02.634182 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:05:02.634183 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:05:02.634847 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:05:02.634848 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:05:02.635371 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:05:02.635373 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:05:02.635748 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:05:02.635749 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:05:02.635827 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:05:02.636296 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:05:02.636300 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:05:02.637526 (+ 1226us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:05:02.639091 (+ 1565us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:05:02.639092 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:05:02.639093 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:05:02.639263 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:05:02.639465 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:05:02.639466 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:05:02.639466 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:05:02.639524 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:05:02.639782 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:05:02.639784 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:05:02.639830 (+ 46us) client_negotiation.cc:769] Sending connection context
0929 19:05:02.639906 (+ 76us) client_negotiation.cc:240] Negotiation successful
0929 19:05:02.640023 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19}
I20260929 19:05:02.640118 22753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:02.633730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52164 (local address 127.0.98.61:43091)
0929 19:05:02.633787 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0929 19:05:02.633789 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:05:02.633800 (+ 11us) server_negotiation.cc:407] Connection header received
0929 19:05:02.633868 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:05:02.633869 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:05:02.633881 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:05:02.633946 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:05:02.634268 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:02.634760 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:02.635454 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:02.635675 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:02.635750 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:05:02.637620 (+ 1870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:05:02.637621 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:05:02.637621 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:05:02.637633 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:05:02.639000 (+ 1367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:05:02.639343 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:05:02.639344 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:05:02.639345 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:05:02.639400 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:05:02.639591 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:05:02.639592 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:05:02.639592 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:05:02.639705 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:05:02.639772 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0929 19:05:02.639931 (+ 159us) server_negotiation.cc:299] Negotiation successful
0929 19:05:02.640018 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260929 19:05:02.640511 22674 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37183 } has no permanent_uuid. Determining permanent_uuid...
I20260929 19:05:02.640866 22676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:02.640750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37183 (local address 127.0.0.1:35208)
0929 19:05:02.640795 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0929 19:05:02.640841 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260929 19:05:02.641172 22676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:02.641066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37183 (local address 127.0.0.1:35210)
0929 19:05:02.641109 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0929 19:05:02.641148 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260929 19:05:02.641250 22674 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:37183: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37183: connect: Connection refused (error 111)
W20260929 19:05:02.670962 22756 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 19:05:02.671216 22756 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:05:02.671247 22756 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:05:02.671298 22756 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260929 19:05:02.672384 22674 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37183 } attempt: 1
W20260929 19:05:02.672636 22756 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 19:05:02.672694 22756 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:05:02.672721 22756 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:05:02.672773 22756 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 19:05:02.672833 22756 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 19:05:02.672863 22676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:02.672691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37183 (local address 127.0.0.1:35220)
0929 19:05:02.672754 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0929 19:05:02.672793 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260929 19:05:02.673236 22676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:02.673150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37183 (local address 127.0.0.1:35224)
0929 19:05:02.673192 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0929 19:05:02.673218 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260929 19:05:02.673322 22674 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:37183: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37183: connect: Connection refused (error 111)
I20260929 19:05:02.674548 22756 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708447806542-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708447806542-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36031,127.0.98.61:43091,127.0.98.60:37183
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708447806542-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:37183
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708447806542-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22756
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708447806542-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 18:09:13 UTC on ccycloud.quasar-hpkalf.root.comops.site
build id 4680913
I20260929 19:05:02.674809 22756 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:05:02.675058 22756 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:05:02.679986 22764 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:05:02.680019 22761 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:05:02.680009 22762 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:05:02.680282 22756 server_base.cc:1034] running on GCE node
I20260929 19:05:02.680512 22756 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:05:02.680922 22756 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:05:02.682088 22756 hybrid_clock.cc:648] HybridClock initialized: now 1790708702682068 us; error 31 us; skew 500 ppm
Sep 29 19:05:02 dist-test-slave-lqkv krb5kdc[22593](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708702, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:05:02.685191 22756 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260929 19:05:02.685753 22756 webserver.cc:460] Webserver started at http://127.0.98.60:40133/ using document root <none> and password file <none>
I20260929 19:05:02.686022 22756 fs_manager.cc:359] Metadata directory not provided
I20260929 19:05:02.686071 22756 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:05:02.686213 22756 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:05:02.687572 22756 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708447806542-392-0/minicluster-data/master-2/data/instance:
uuid: "42a00da358fb43e5af7d11184611c861"
format_stamp: "Formatted at 2026-09-29 19:05:02 on dist-test-slave-lqkv"
server_key: "ec6a57bab4c15c2d4116f73fab4df4a1"
server_key_iv: "ae94803134f10825be13ba1923a7d727"
server_key_version: "encryptionkey@0"
I20260929 19:05:02.687942 22756 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708447806542-392-0/minicluster-data/master-2/wal/instance:
uuid: "42a00da358fb43e5af7d11184611c861"
format_stamp: "Formatted at 2026-09-29 19:05:02 on dist-test-slave-lqkv"
server_key: "ec6a57bab4c15c2d4116f73fab4df4a1"
server_key_iv: "ae94803134f10825be13ba1923a7d727"
server_key_version: "encryptionkey@0"
I20260929 19:05:02.689345 22756 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 19:05:02.690050 22771 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:05:02.690201 22756 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260929 19:05:02.690297 22756 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708447806542-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708447806542-392-0/minicluster-data/master-2/wal
uuid: "42a00da358fb43e5af7d11184611c861"
format_stamp: "Formatted at 2026-09-29 19:05:02 on dist-test-slave-lqkv"
server_key: "ec6a57bab4c15c2d4116f73fab4df4a1"
server_key_iv: "ae94803134f10825be13ba1923a7d727"
server_key_version: "encryptionkey@0"
I20260929 19:05:02.690367 22756 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708447806542-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708447806542-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708447806542-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:05:02.697748 22749 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37183 } attempt: 2
I20260929 19:05:02.697844 22756 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:05:02.698228 22751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:02.698073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37183 (local address 127.0.0.1:35240)
0929 19:05:02.698140 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0929 19:05:02.698174 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260929 19:05:02.698575 22751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:02.698475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37183 (local address 127.0.0.1:35250)
0929 19:05:02.698521 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0929 19:05:02.698551 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260929 19:05:02.698678 22749 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:37183: Network error: Client connection negotiation failed: client connection to 127.0.98.60:37183: connect: Connection refused (error 111)
I20260929 19:05:02.699831 22756 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:05:02.699935 22756 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:05:02.703719 22756 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:37183
I20260929 19:05:02.703835 22823 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:37183 every 8 connection(s)
I20260929 19:05:02.704120 22756 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708447806542-392-0/minicluster-data/master-2/data/info.pb
I20260929 19:05:02.705040 22824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:05:02.706573 22824 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36031 } has no permanent_uuid. Determining permanent_uuid...
I20260929 19:05:02.712244 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUbnLHE/build/release/bin/kudu as pid 22756
Sep 29 19:05:02 dist-test-slave-lqkv krb5kdc[22593](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708702, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 19:05:02.712321 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790708447806542-392-0/minicluster-data/master-2/wal/instance
I20260929 19:05:02.712524 392 external_mini_cluster.cc:1428] Setting key c6407d909eeb76076b3cdd158167de8b
I20260929 19:05:02.715828 22677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:02.707314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60742 (local address 127.0.98.62:36031)
0929 19:05:02.707371 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0929 19:05:02.707373 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:05:02.707607 (+ 234us) server_negotiation.cc:407] Connection header received
0929 19:05:02.708067 (+ 460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:05:02.708068 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:05:02.708079 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:05:02.708146 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:05:02.708822 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:02.709345 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:02.710344 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:02.710536 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:02.710599 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:05:02.712785 (+ 2186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:05:02.712787 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:05:02.712787 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:05:02.712795 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:05:02.714446 (+ 1651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:05:02.714900 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:05:02.714901 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:05:02.714901 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:05:02.714965 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:05:02.715196 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:05:02.715197 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:05:02.715197 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:05:02.715334 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:05:02.715401 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0929 19:05:02.715659 (+ 258us) server_negotiation.cc:299] Negotiation successful
0929 19:05:02.715746 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260929 19:05:02.716037 22826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:02.707273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36031 (local address 127.0.0.1:60742)
0929 19:05:02.707458 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0929 19:05:02.707501 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 19:05:02.707962 (+ 461us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:05:02.708237 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:05:02.708241 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:05:02.708489 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:05:02.708681 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:05:02.708685 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:05:02.709462 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:05:02.709462 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:05:02.710257 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:05:02.710260 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:05:02.710620 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:05:02.710621 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:05:02.710725 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:05:02.711232 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:05:02.711240 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:05:02.712693 (+ 1453us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:05:02.714522 (+ 1829us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:05:02.714527 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:05:02.714529 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:05:02.714816 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:05:02.715032 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:05:02.715033 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:05:02.715033 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:05:02.715133 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:05:02.715409 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:05:02.715415 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:05:02.715533 (+ 118us) client_negotiation.cc:769] Sending connection context
0929 19:05:02.715622 (+ 89us) client_negotiation.cc:240] Negotiation successful
0929 19:05:02.715774 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":109,"threads_started":1}
I20260929 19:05:02.716882 22824 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43091 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 19:05:02 dist-test-slave-lqkv krb5kdc[22593](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708702, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 29 19:05:02 dist-test-slave-lqkv krb5kdc[22593](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708702, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260929 19:05:02.721926 22828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:02.712910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35262 (local address 127.0.98.60:37183)
0929 19:05:02.713100 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0929 19:05:02.713105 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 19:05:02.713130 (+ 25us) server_negotiation.cc:407] Connection header received
0929 19:05:02.713189 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:05:02.713191 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:05:02.713205 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:05:02.713288 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:05:02.713973 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:02.714821 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:02.715757 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:02.716227 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:02.716296 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:05:02.718977 (+ 2681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:05:02.718985 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:05:02.718985 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:05:02.718992 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:05:02.720676 (+ 1684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:05:02.721085 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:05:02.721089 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:05:02.721089 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:05:02.721152 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:05:02.721352 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:05:02.721352 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:05:02.721353 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:05:02.721497 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:05:02.721653 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0929 19:05:02.721732 (+ 79us) server_negotiation.cc:299] Negotiation successful
0929 19:05:02.721809 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":87,"threads_started":1}
I20260929 19:05:02.723629 22826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:02.717176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43091 (local address 127.0.0.1:52168)
0929 19:05:02.717228 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0929 19:05:02.717239 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 19:05:02.717282 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:05:02.717404 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:05:02.717405 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:05:02.717553 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:05:02.717598 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:05:02.717600 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:05:02.718396 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:05:02.718396 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:05:02.719038 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:05:02.719040 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:05:02.719399 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:05:02.719400 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:05:02.719470 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:05:02.719823 (+ 353us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:05:02.719827 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:05:02.720943 (+ 1116us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:05:02.722629 (+ 1686us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:05:02.722630 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:05:02.722630 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:05:02.722853 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:05:02.723041 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:05:02.723042 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:05:02.723043 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:05:02.723108 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:05:02.723315 (+ 207us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:05:02.723316 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:05:02.723363 (+ 47us) client_negotiation.cc:769] Sending connection context
0929 19:05:02.723442 (+ 79us) client_negotiation.cc:240] Negotiation successful
0929 19:05:02.723540 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260929 19:05:02.723654 22753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:02.717252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52168 (local address 127.0.98.61:43091)
0929 19:05:02.717298 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0929 19:05:02.717300 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:05:02.717309 (+ 9us) server_negotiation.cc:407] Connection header received
0929 19:05:02.717335 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:05:02.717336 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:05:02.717345 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:05:02.717400 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:05:02.717675 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:02.718319 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:02.719124 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:02.719316 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:02.719389 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:05:02.721026 (+ 1637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:05:02.721027 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:05:02.721028 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:05:02.721035 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:05:02.722566 (+ 1531us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:05:02.722916 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:05:02.722917 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:05:02.722918 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:05:02.722978 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:05:02.723165 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:05:02.723166 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:05:02.723166 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:05:02.723259 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:05:02.723302 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0929 19:05:02.723458 (+ 156us) server_negotiation.cc:299] Negotiation successful
0929 19:05:02.723569 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260929 19:05:02.724279 22824 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37183 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 19:05:02 dist-test-slave-lqkv krb5kdc[22593](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708702, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 29 19:05:02 dist-test-slave-lqkv krb5kdc[22593](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708702, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260929 19:05:02.730073 22677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:02.722714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60744 (local address 127.0.98.62:36031)
0929 19:05:02.722768 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0929 19:05:02.722770 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:05:02.722778 (+ 8us) server_negotiation.cc:407] Connection header received
0929 19:05:02.722873 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:05:02.722874 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:05:02.722899 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:05:02.722950 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:05:02.723556 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:02.724098 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:02.725009 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:02.725292 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:02.725361 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:05:02.727424 (+ 2063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:05:02.727425 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:05:02.727425 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:05:02.727432 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:05:02.728961 (+ 1529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:05:02.729281 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:05:02.729282 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:05:02.729282 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:05:02.729335 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:05:02.729542 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:05:02.729543 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:05:02.729543 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:05:02.729673 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:05:02.729858 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0929 19:05:02.729897 (+ 39us) server_negotiation.cc:299] Negotiation successful
0929 19:05:02.729970 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260929 19:05:02.730736 22828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:02.724611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35274 (local address 127.0.98.60:37183)
0929 19:05:02.724661 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0929 19:05:02.724663 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:05:02.724682 (+ 19us) server_negotiation.cc:407] Connection header received
0929 19:05:02.724697 (+ 15us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:05:02.724698 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:05:02.724707 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:05:02.724745 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:05:02.725034 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:02.725625 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:02.726236 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:02.726443 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:02.726527 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:05:02.728233 (+ 1706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:05:02.728234 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:05:02.728235 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:05:02.728242 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:05:02.729604 (+ 1362us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:05:02.729945 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:05:02.729946 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:05:02.729947 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:05:02.730012 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:05:02.730202 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:05:02.730202 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:05:02.730203 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:05:02.730308 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:05:02.730366 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0929 19:05:02.730520 (+ 154us) server_negotiation.cc:299] Negotiation successful
0929 19:05:02.730636 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":55,"server-negotiator.queue_time_us":34}
I20260929 19:05:02.730737 22826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:02.724546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37183 (local address 127.0.0.1:35274)
0929 19:05:02.724578 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0929 19:05:02.724588 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 19:05:02.724621 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:05:02.724756 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:05:02.724757 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:05:02.724923 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:05:02.724948 (+ 25us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:05:02.724950 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:05:02.725708 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:05:02.725709 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:05:02.726163 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:05:02.726164 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:05:02.726527 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:05:02.726527 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:05:02.726574 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:05:02.726958 (+ 384us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:05:02.726963 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:05:02.728151 (+ 1188us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:05:02.729672 (+ 1521us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:05:02.729673 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:05:02.729673 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:05:02.729864 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:05:02.730076 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:05:02.730077 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:05:02.730077 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:05:02.730132 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:05:02.730374 (+ 242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:05:02.730375 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:05:02.730424 (+ 49us) client_negotiation.cc:769] Sending connection context
0929 19:05:02.730490 (+ 66us) client_negotiation.cc:240] Negotiation successful
0929 19:05:02.730596 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17}
I20260929 19:05:02.732072 22824 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42a00da358fb43e5af7d11184611c861: Bootstrap starting.
I20260929 19:05:02.732965 22824 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 42a00da358fb43e5af7d11184611c861: Neither blocks nor log segments found. Creating new log.
I20260929 19:05:02.733271 22824 log.cc:824] T 00000000000000000000000000000000 P 42a00da358fb43e5af7d11184611c861: Log is configured to *not* fsync() on all Append() calls
I20260929 19:05:02.734045 22824 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42a00da358fb43e5af7d11184611c861: No bootstrap required, opened a new log
Sep 29 19:05:02 dist-test-slave-lqkv krb5kdc[22593](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708702, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260929 19:05:02.736137 22824 raft_consensus.cc:348] T 00000000000000000000000000000000 P 42a00da358fb43e5af7d11184611c861 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315bce7e99d447a78fc04e183250a8a6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36031 } } peers { permanent_uuid: "d033f97a05654c89a2c695a4d3f638a7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43091 } } peers { permanent_uuid: "42a00da358fb43e5af7d11184611c861" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37183 } }
I20260929 19:05:02.736315 22824 raft_consensus.cc:729] T 00000000000000000000000000000000 P 42a00da358fb43e5af7d11184611c861 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42a00da358fb43e5af7d11184611c861, State: Initialized, Role: FOLLOWER
I20260929 19:05:02.736415 22824 consensus_queue.cc:260] T 00000000000000000000000000000000 P 42a00da358fb43e5af7d11184611c861 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315bce7e99d447a78fc04e183250a8a6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36031 } } peers { permanent_uuid: "d033f97a05654c89a2c695a4d3f638a7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43091 } } peers { permanent_uuid: "42a00da358fb43e5af7d11184611c861" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37183 } }
I20260929 19:05:02.736744 22824 sys_catalog.cc:564] T 00000000000000000000000000000000 P 42a00da358fb43e5af7d11184611c861 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 19:05:02.736720 22830 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42a00da358fb43e5af7d11184611c861 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315bce7e99d447a78fc04e183250a8a6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36031 } } peers { permanent_uuid: "d033f97a05654c89a2c695a4d3f638a7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43091 } } peers { permanent_uuid: "42a00da358fb43e5af7d11184611c861" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37183 } } }
I20260929 19:05:02.736861 22830 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42a00da358fb43e5af7d11184611c861 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 19:05:02.738415 22753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:02.730719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52170 (local address 127.0.98.61:43091)
0929 19:05:02.730758 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0929 19:05:02.730761 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 19:05:02.730836 (+ 75us) server_negotiation.cc:407] Connection header received
0929 19:05:02.730911 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:05:02.730912 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:05:02.730922 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:05:02.730968 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:05:02.731511 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:02.731964 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:02.732672 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:02.732956 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:02.733026 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:05:02.735531 (+ 2505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:05:02.735532 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:05:02.735532 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:05:02.735538 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:05:02.737125 (+ 1587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:05:02.737560 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:05:02.737561 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:05:02.737561 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:05:02.737608 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:05:02.737831 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:05:02.737832 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:05:02.737833 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:05:02.737986 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:05:02.738145 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0929 19:05:02.738245 (+ 100us) server_negotiation.cc:299] Negotiation successful
0929 19:05:02.738319 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
W20260929 19:05:02.738660 22841 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 42a00da358fb43e5af7d11184611c861: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 19:05:02.738722 22841 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 19:05:02.739903 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 19:05:02.739194 22756 master_runner.cc:186] Error getting master registration for 127.0.98.62:36031: OK, instance_id { permanent_uuid: "315bce7e99d447a78fc04e183250a8a6" instance_seqno: 1790708702407294 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 19:05:02.741760 22756 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 19:05:02.741132 22648 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "315bce7e99d447a78fc04e183250a8a6"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60744
I20260929 19:05:02.748461 22674 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37183 } attempt: 2
Sep 29 19:05:02 dist-test-slave-lqkv krb5kdc[22593](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708702, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260929 19:05:02.756377 22828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:02.748930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35278 (local address 127.0.98.60:37183)
0929 19:05:02.748986 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0929 19:05:02.748988 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:05:02.749013 (+ 25us) server_negotiation.cc:407] Connection header received
0929 19:05:02.749037 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:05:02.749038 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:05:02.749047 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:05:02.749116 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:05:02.749469 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:02.750201 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:02.751181 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:02.751466 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:02.751530 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:05:02.753708 (+ 2178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:05:02.753710 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:05:02.753711 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:05:02.753718 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:05:02.755121 (+ 1403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:05:02.755509 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:05:02.755511 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:05:02.755511 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:05:02.755567 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:05:02.755750 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:05:02.755751 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:05:02.755752 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:05:02.755959 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:05:02.756029 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 19:05:02.756219 (+ 190us) server_negotiation.cc:299] Negotiation successful
0929 19:05:02.756291 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260929 19:05:02.756377 22676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:02.748843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37183 (local address 127.0.0.1:35278)
0929 19:05:02.748899 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0929 19:05:02.748911 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0929 19:05:02.748956 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:05:02.749141 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:05:02.749142 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:05:02.749334 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:05:02.749387 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:05:02.749389 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:05:02.750288 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:05:02.750290 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:05:02.751091 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:05:02.751094 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:05:02.751540 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:05:02.751541 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:05:02.751601 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:05:02.752063 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:05:02.752068 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:05:02.753618 (+ 1550us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:05:02.755225 (+ 1607us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:05:02.755226 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:05:02.755227 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:05:02.755444 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:05:02.755640 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:05:02.755641 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:05:02.755641 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:05:02.755700 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:05:02.756057 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:05:02.756058 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:05:02.756104 (+ 46us) client_negotiation.cc:769] Sending connection context
0929 19:05:02.756166 (+ 62us) client_negotiation.cc:240] Negotiation successful
0929 19:05:02.756271 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260929 19:05:02.757747 22674 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 315bce7e99d447a78fc04e183250a8a6: Bootstrap starting.
I20260929 19:05:02.758380 22674 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 315bce7e99d447a78fc04e183250a8a6: Neither blocks nor log segments found. Creating new log.
I20260929 19:05:02.758667 22674 log.cc:824] T 00000000000000000000000000000000 P 315bce7e99d447a78fc04e183250a8a6: Log is configured to *not* fsync() on all Append() calls
I20260929 19:05:02.759523 22674 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 315bce7e99d447a78fc04e183250a8a6: No bootstrap required, opened a new log
I20260929 19:05:02.760943 22674 raft_consensus.cc:348] T 00000000000000000000000000000000 P 315bce7e99d447a78fc04e183250a8a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315bce7e99d447a78fc04e183250a8a6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36031 } } peers { permanent_uuid: "d033f97a05654c89a2c695a4d3f638a7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43091 } } peers { permanent_uuid: "42a00da358fb43e5af7d11184611c861" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37183 } }
I20260929 19:05:02.761093 22674 raft_consensus.cc:729] T 00000000000000000000000000000000 P 315bce7e99d447a78fc04e183250a8a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 315bce7e99d447a78fc04e183250a8a6, State: Initialized, Role: FOLLOWER
I20260929 19:05:02.761184 22674 consensus_queue.cc:260] T 00000000000000000000000000000000 P 315bce7e99d447a78fc04e183250a8a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315bce7e99d447a78fc04e183250a8a6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36031 } } peers { permanent_uuid: "d033f97a05654c89a2c695a4d3f638a7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43091 } } peers { permanent_uuid: "42a00da358fb43e5af7d11184611c861" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37183 } }
I20260929 19:05:02.761525 22674 sys_catalog.cc:564] T 00000000000000000000000000000000 P 315bce7e99d447a78fc04e183250a8a6 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 19:05:02.761494 22844 sys_catalog.cc:455] T 00000000000000000000000000000000 P 315bce7e99d447a78fc04e183250a8a6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315bce7e99d447a78fc04e183250a8a6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36031 } } peers { permanent_uuid: "d033f97a05654c89a2c695a4d3f638a7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43091 } } peers { permanent_uuid: "42a00da358fb43e5af7d11184611c861" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37183 } } }
I20260929 19:05:02.761595 22648 raft_consensus.cc:482] T 00000000000000000000000000000000 P 315bce7e99d447a78fc04e183250a8a6 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 19:05:02.761690 22844 sys_catalog.cc:458] T 00000000000000000000000000000000 P 315bce7e99d447a78fc04e183250a8a6 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 19:05:02.761727 22648 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 315bce7e99d447a78fc04e183250a8a6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:05:02.762684 22648 raft_consensus.cc:504] T 00000000000000000000000000000000 P 315bce7e99d447a78fc04e183250a8a6 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315bce7e99d447a78fc04e183250a8a6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36031 } } peers { permanent_uuid: "d033f97a05654c89a2c695a4d3f638a7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43091 } } peers { permanent_uuid: "42a00da358fb43e5af7d11184611c861" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37183 } }
I20260929 19:05:02.762954 22648 leader_election.cc:288] T 00000000000000000000000000000000 P 315bce7e99d447a78fc04e183250a8a6 [CANDIDATE]: Term 1 election: Requested vote from peers d033f97a05654c89a2c695a4d3f638a7 (127.0.98.61:43091), 42a00da358fb43e5af7d11184611c861 (127.0.98.60:37183)
I20260929 19:05:02.763695 22798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "315bce7e99d447a78fc04e183250a8a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "42a00da358fb43e5af7d11184611c861"
I20260929 19:05:02.763835 22798 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 42a00da358fb43e5af7d11184611c861 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:05:02.764822 22798 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 42a00da358fb43e5af7d11184611c861 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 315bce7e99d447a78fc04e183250a8a6 in term 1.
I20260929 19:05:02.765090 22625 leader_election.cc:302] T 00000000000000000000000000000000 P 315bce7e99d447a78fc04e183250a8a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 315bce7e99d447a78fc04e183250a8a6, 42a00da358fb43e5af7d11184611c861; no voters:
I20260929 19:05:02.765213 22844 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 315bce7e99d447a78fc04e183250a8a6 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:05:02.765363 22844 raft_consensus.cc:686] T 00000000000000000000000000000000 P 315bce7e99d447a78fc04e183250a8a6 [term 1 LEADER]: Becoming Leader. State: Replica: 315bce7e99d447a78fc04e183250a8a6, State: Running, Role: LEADER
I20260929 19:05:02.765456 22844 consensus_queue.cc:237] T 00000000000000000000000000000000 P 315bce7e99d447a78fc04e183250a8a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315bce7e99d447a78fc04e183250a8a6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36031 } } peers { permanent_uuid: "d033f97a05654c89a2c695a4d3f638a7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43091 } } peers { permanent_uuid: "42a00da358fb43e5af7d11184611c861" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37183 } }
I20260929 19:05:02.766124 22860 sys_catalog.cc:455] T 00000000000000000000000000000000 P 315bce7e99d447a78fc04e183250a8a6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 315bce7e99d447a78fc04e183250a8a6. Latest consensus state: current_term: 1 leader_uuid: "315bce7e99d447a78fc04e183250a8a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315bce7e99d447a78fc04e183250a8a6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36031 } } peers { permanent_uuid: "d033f97a05654c89a2c695a4d3f638a7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43091 } } peers { permanent_uuid: "42a00da358fb43e5af7d11184611c861" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37183 } } }
I20260929 19:05:02.766691 22860 sys_catalog.cc:458] T 00000000000000000000000000000000 P 315bce7e99d447a78fc04e183250a8a6 [sys.catalog]: This master's current role is: LEADER
W20260929 19:05:02.766899 22855 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 315bce7e99d447a78fc04e183250a8a6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260929 19:05:02.764249 22606 master_runner.cc:186] Error getting master registration for 127.0.98.61:43091: OK, instance_id { permanent_uuid: "d033f97a05654c89a2c695a4d3f638a7" instance_seqno: 1790708702554321 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 19:05:02.767045 22606 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260929 19:05:02.767160 22855 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 19:05:02.767361 22863 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 19:05:02.767557 22863 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 19:05:02.769202 22798 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 42a00da358fb43e5af7d11184611c861 [term 1 FOLLOWER]: Refusing update from remote peer 315bce7e99d447a78fc04e183250a8a6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 19:05:02.769518 22860 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 315bce7e99d447a78fc04e183250a8a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42a00da358fb43e5af7d11184611c861" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 19:05:02.770771 22830 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42a00da358fb43e5af7d11184611c861 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 315bce7e99d447a78fc04e183250a8a6. Latest consensus state: current_term: 1 leader_uuid: "315bce7e99d447a78fc04e183250a8a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315bce7e99d447a78fc04e183250a8a6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36031 } } peers { permanent_uuid: "d033f97a05654c89a2c695a4d3f638a7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43091 } } peers { permanent_uuid: "42a00da358fb43e5af7d11184611c861" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37183 } } }
I20260929 19:05:02.770874 22830 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42a00da358fb43e5af7d11184611c861 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 19:05:02.771986 22860 sys_catalog.cc:455] T 00000000000000000000000000000000 P 315bce7e99d447a78fc04e183250a8a6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "315bce7e99d447a78fc04e183250a8a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315bce7e99d447a78fc04e183250a8a6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36031 } } peers { permanent_uuid: "d033f97a05654c89a2c695a4d3f638a7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43091 } } peers { permanent_uuid: "42a00da358fb43e5af7d11184611c861" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37183 } } }
I20260929 19:05:02.772063 22860 sys_catalog.cc:458] T 00000000000000000000000000000000 P 315bce7e99d447a78fc04e183250a8a6 [sys.catalog]: This master's current role is: LEADER
I20260929 19:05:02.772397 22830 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42a00da358fb43e5af7d11184611c861 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "315bce7e99d447a78fc04e183250a8a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315bce7e99d447a78fc04e183250a8a6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36031 } } peers { permanent_uuid: "d033f97a05654c89a2c695a4d3f638a7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43091 } } peers { permanent_uuid: "42a00da358fb43e5af7d11184611c861" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37183 } } }
I20260929 19:05:02.772481 22830 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42a00da358fb43e5af7d11184611c861 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 19:05:02.772680 22867 mvcc.cc:205] Tried to move back new op lower bound from 7334742846539976704 to 7334742846527942656. Current Snapshot: MvccSnapshot[applied={T|T < 7334742846539976704}]
I20260929 19:05:02.773171 22868 mvcc.cc:205] Tried to move back new op lower bound from 7334742846539976704 to 7334742846527942656. Current Snapshot: MvccSnapshot[applied={T|T < 7334742846539976704}]
I20260929 19:05:02.777594 22863 catalog_manager.cc:1347] Generated new cluster ID: ffc692d784e1458080711a14cefda39a
I20260929 19:05:02.777654 22863 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 19:05:02.784307 22677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:02.767189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60752 (local address 127.0.98.62:36031)
0929 19:05:02.767231 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0929 19:05:02.767233 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:05:02.768375 (+ 1142us) server_negotiation.cc:407] Connection header received
0929 19:05:02.769250 (+ 875us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:05:02.769251 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:05:02.769261 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:05:02.769291 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:05:02.771780 (+ 2489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:02.772464 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:02.775327 (+ 2863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:02.775638 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:02.775705 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:05:02.779170 (+ 3465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:05:02.779171 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:05:02.779172 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:05:02.779179 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:05:02.780894 (+ 1715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:05:02.782158 (+ 1264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:05:02.782160 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:05:02.782160 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:05:02.782471 (+ 311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:05:02.782695 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:05:02.782696 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:05:02.782697 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:05:02.783805 (+ 1108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:05:02.783900 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0929 19:05:02.784110 (+ 210us) server_negotiation.cc:299] Negotiation successful
0929 19:05:02.784181 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260929 19:05:02.784646 22753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:02.767403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52178 (local address 127.0.98.61:43091)
0929 19:05:02.767446 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0929 19:05:02.767448 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:05:02.768095 (+ 647us) server_negotiation.cc:407] Connection header received
0929 19:05:02.768199 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:05:02.768200 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:05:02.768211 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:05:02.768243 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:05:02.770055 (+ 1812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:02.770744 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:02.774411 (+ 3667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:02.774724 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:02.774767 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:05:02.776422 (+ 1655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:05:02.776423 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:05:02.776424 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:05:02.776431 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:05:02.778155 (+ 1724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:05:02.780247 (+ 2092us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:05:02.780249 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:05:02.780249 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:05:02.780318 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:05:02.780869 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:05:02.780870 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:05:02.780871 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:05:02.781026 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:05:02.781115 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0929 19:05:02.784468 (+ 3353us) server_negotiation.cc:299] Negotiation successful
0929 19:05:02.784557 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260929 19:05:02.786288 22828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:02.767745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35286 (local address 127.0.98.60:37183)
0929 19:05:02.767780 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0929 19:05:02.767782 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:05:02.773586 (+ 5804us) server_negotiation.cc:407] Connection header received
0929 19:05:02.773634 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:05:02.773635 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:05:02.773645 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:05:02.773674 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:05:02.774365 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:02.775029 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:02.778455 (+ 3426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:02.778836 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:02.778909 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:05:02.781941 (+ 3032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:05:02.781955 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:05:02.781956 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:05:02.781964 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:05:02.783617 (+ 1653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:05:02.785287 (+ 1670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:05:02.785289 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:05:02.785289 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:05:02.785350 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:05:02.785628 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:05:02.785642 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:05:02.785643 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:05:02.785773 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:05:02.785875 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 19:05:02.786094 (+ 219us) server_negotiation.cc:299] Negotiation successful
0929 19:05:02.786176 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260929 19:05:02.789813 22749 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37183 } attempt: 3
Sep 29 19:05:02 dist-test-slave-lqkv krb5kdc[22593](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708702, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260929 19:05:02.798116 22751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:02.790070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:37183 (local address 127.0.0.1:35288)
0929 19:05:02.790145 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0929 19:05:02.790156 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0929 19:05:02.790195 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:05:02.790450 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:05:02.790451 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:05:02.790595 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:05:02.790632 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:05:02.790634 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:05:02.791525 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:05:02.791526 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:05:02.792036 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:05:02.792037 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:05:02.792509 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:05:02.792510 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:05:02.792575 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:05:02.793059 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:05:02.793064 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:05:02.794653 (+ 1589us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:05:02.796979 (+ 2326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:05:02.796980 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:05:02.796981 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:05:02.797211 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:05:02.797418 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:05:02.797418 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:05:02.797419 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:05:02.797471 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:05:02.797781 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:05:02.797782 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:05:02.797843 (+ 61us) client_negotiation.cc:769] Sending connection context
0929 19:05:02.797908 (+ 65us) client_negotiation.cc:240] Negotiation successful
0929 19:05:02.798011 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260929 19:05:02.798302 22828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:02.790284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35288 (local address 127.0.98.60:37183)
0929 19:05:02.790333 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0929 19:05:02.790335 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:05:02.790341 (+ 6us) server_negotiation.cc:407] Connection header received
0929 19:05:02.790372 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:05:02.790373 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:05:02.790384 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:05:02.790432 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:05:02.790706 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:02.791435 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:02.792117 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:02.792442 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:02.792510 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:05:02.795083 (+ 2573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:05:02.795084 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:05:02.795085 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:05:02.795093 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:05:02.796861 (+ 1768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:05:02.797284 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:05:02.797285 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:05:02.797286 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:05:02.797358 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:05:02.797548 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:05:02.797549 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:05:02.797549 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:05:02.797679 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:05:02.797765 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 19:05:02.797939 (+ 174us) server_negotiation.cc:299] Negotiation successful
0929 19:05:02.798065 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260929 19:05:02.799883 22749 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d033f97a05654c89a2c695a4d3f638a7: Bootstrap starting.
I20260929 19:05:02.800653 22749 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d033f97a05654c89a2c695a4d3f638a7: Neither blocks nor log segments found. Creating new log.
I20260929 19:05:02.801334 22749 log.cc:824] T 00000000000000000000000000000000 P d033f97a05654c89a2c695a4d3f638a7: Log is configured to *not* fsync() on all Append() calls
I20260929 19:05:02.802266 22749 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d033f97a05654c89a2c695a4d3f638a7: No bootstrap required, opened a new log
I20260929 19:05:02.804314 22749 raft_consensus.cc:348] T 00000000000000000000000000000000 P d033f97a05654c89a2c695a4d3f638a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315bce7e99d447a78fc04e183250a8a6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36031 } } peers { permanent_uuid: "d033f97a05654c89a2c695a4d3f638a7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43091 } } peers { permanent_uuid: "42a00da358fb43e5af7d11184611c861" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37183 } }
I20260929 19:05:02.804455 22749 raft_consensus.cc:729] T 00000000000000000000000000000000 P d033f97a05654c89a2c695a4d3f638a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d033f97a05654c89a2c695a4d3f638a7, State: Initialized, Role: FOLLOWER
I20260929 19:05:02.804550 22749 consensus_queue.cc:260] T 00000000000000000000000000000000 P d033f97a05654c89a2c695a4d3f638a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315bce7e99d447a78fc04e183250a8a6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36031 } } peers { permanent_uuid: "d033f97a05654c89a2c695a4d3f638a7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43091 } } peers { permanent_uuid: "42a00da358fb43e5af7d11184611c861" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37183 } }
I20260929 19:05:02.804932 22749 sys_catalog.cc:564] T 00000000000000000000000000000000 P d033f97a05654c89a2c695a4d3f638a7 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 19:05:02.805027 22722 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d033f97a05654c89a2c695a4d3f638a7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:05:02.806020 22722 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d033f97a05654c89a2c695a4d3f638a7 [term 1 FOLLOWER]: Refusing update from remote peer 315bce7e99d447a78fc04e183250a8a6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 19:05:02.806214 22723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "315bce7e99d447a78fc04e183250a8a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "d033f97a05654c89a2c695a4d3f638a7"
I20260929 19:05:02.806541 22844 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 315bce7e99d447a78fc04e183250a8a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d033f97a05654c89a2c695a4d3f638a7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 19:05:02.804914 22873 sys_catalog.cc:455] T 00000000000000000000000000000000 P d033f97a05654c89a2c695a4d3f638a7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315bce7e99d447a78fc04e183250a8a6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36031 } } peers { permanent_uuid: "d033f97a05654c89a2c695a4d3f638a7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43091 } } peers { permanent_uuid: "42a00da358fb43e5af7d11184611c861" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37183 } } }
I20260929 19:05:02.807261 22723 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d033f97a05654c89a2c695a4d3f638a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 315bce7e99d447a78fc04e183250a8a6 in term 1.
W20260929 19:05:02.808387 22884 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d033f97a05654c89a2c695a4d3f638a7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 19:05:02.808437 22884 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 19:05:02.809852 22873 sys_catalog.cc:458] T 00000000000000000000000000000000 P d033f97a05654c89a2c695a4d3f638a7 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 19:05:02.809968 22873 sys_catalog.cc:455] T 00000000000000000000000000000000 P d033f97a05654c89a2c695a4d3f638a7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 315bce7e99d447a78fc04e183250a8a6. Latest consensus state: current_term: 1 leader_uuid: "315bce7e99d447a78fc04e183250a8a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315bce7e99d447a78fc04e183250a8a6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36031 } } peers { permanent_uuid: "d033f97a05654c89a2c695a4d3f638a7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43091 } } peers { permanent_uuid: "42a00da358fb43e5af7d11184611c861" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37183 } } }
I20260929 19:05:02.810091 22873 sys_catalog.cc:458] T 00000000000000000000000000000000 P d033f97a05654c89a2c695a4d3f638a7 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 19:05:02.810171 22873 sys_catalog.cc:455] T 00000000000000000000000000000000 P d033f97a05654c89a2c695a4d3f638a7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "315bce7e99d447a78fc04e183250a8a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315bce7e99d447a78fc04e183250a8a6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36031 } } peers { permanent_uuid: "d033f97a05654c89a2c695a4d3f638a7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43091 } } peers { permanent_uuid: "42a00da358fb43e5af7d11184611c861" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37183 } } }
I20260929 19:05:02.810281 22873 sys_catalog.cc:458] T 00000000000000000000000000000000 P d033f97a05654c89a2c695a4d3f638a7 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 19:05:02.811223 22887 mvcc.cc:205] Tried to move back new op lower bound from 7334742846539976704 to 7334742846527942656. Current Snapshot: MvccSnapshot[applied={T|T < 7334742846539976704}]
I20260929 19:05:02.829677 22844 sys_catalog.cc:455] T 00000000000000000000000000000000 P 315bce7e99d447a78fc04e183250a8a6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "315bce7e99d447a78fc04e183250a8a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315bce7e99d447a78fc04e183250a8a6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36031 } } peers { permanent_uuid: "d033f97a05654c89a2c695a4d3f638a7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43091 } } peers { permanent_uuid: "42a00da358fb43e5af7d11184611c861" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 37183 } } }
I20260929 19:05:02.829809 22844 sys_catalog.cc:458] T 00000000000000000000000000000000 P 315bce7e99d447a78fc04e183250a8a6 [sys.catalog]: This master's current role is: LEADER
I20260929 19:05:02.832233 22863 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 19:05:02.833328 22863 catalog_manager.cc:1504] Loading token signing keys...
I20260929 19:05:02.841130 22863 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 315bce7e99d447a78fc04e183250a8a6: Generated new TSK 0
I20260929 19:05:02.841331 22863 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 19:05:02.864598 22753 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:05:02.862937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52190 (local address 127.0.98.61:43091)
0929 19:05:02.862984 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0929 19:05:02.862986 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:05:02.863894 (+ 908us) server_negotiation.cc:407] Connection header received
0929 19:05:02.864001 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:05:02.864002 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:05:02.864016 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:05:02.864053 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:05:02.864526 (+ 473us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52190: BlockingRecv error: recv got EOF from 127.0.0.1:52190 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
W20260929 19:05:02.865370 22828 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:05:02.863304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35290 (local address 127.0.98.60:37183)
0929 19:05:02.863345 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0929 19:05:02.863347 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:05:02.864853 (+ 1506us) server_negotiation.cc:407] Connection header received
0929 19:05:02.864949 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:05:02.864950 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:05:02.864960 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:05:02.864987 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:05:02.865313 (+ 326us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35290: BlockingRecv error: recv got EOF from 127.0.0.1:35290 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260929 19:05:02.866108 22677 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:05:02.863638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60756 (local address 127.0.98.62:36031)
0929 19:05:02.863677 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0929 19:05:02.863679 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:05:02.865596 (+ 1917us) server_negotiation.cc:407] Connection header received
0929 19:05:02.865688 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:05:02.865690 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:05:02.865703 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:05:02.865734 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:05:02.866029 (+ 295us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60756: BlockingRecv error: recv got EOF from 127.0.0.1:60756 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260929 19:05:02.884115 22753 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:05:02.877682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52204 (local address 127.0.98.61:43091)
0929 19:05:02.877736 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0929 19:05:02.877739 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 19:05:02.883371 (+ 5632us) server_negotiation.cc:407] Connection header received
0929 19:05:02.883518 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:05:02.883519 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:05:02.883544 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:05:02.883592 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:05:02.884051 (+ 459us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52204: BlockingRecv error: recv got EOF from 127.0.0.1:52204 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
W20260929 19:05:02.887549 22753 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:05:02.886667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52206 (local address 127.0.98.61:43091)
0929 19:05:02.886707 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0929 19:05:02.886708 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 19:05:02.886968 (+ 260us) server_negotiation.cc:407] Connection header received
0929 19:05:02.887071 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:05:02.887072 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:05:02.887083 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:05:02.887115 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:05:02.887496 (+ 381us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52206: BlockingRecv error: recv got EOF from 127.0.0.1:52206 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260929 19:05:02.891649 22753 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:05:02.890806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52208 (local address 127.0.98.61:43091)
0929 19:05:02.890842 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0929 19:05:02.890844 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:05:02.891089 (+ 245us) server_negotiation.cc:407] Connection header received
0929 19:05:02.891193 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:05:02.891194 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:05:02.891205 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:05:02.891234 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:05:02.891597 (+ 363us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52208: BlockingRecv error: recv got EOF from 127.0.0.1:52208 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260929 19:05:02.902266 22753 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:05:02.901283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52218 (local address 127.0.98.61:43091)
0929 19:05:02.901329 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0929 19:05:02.901331 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:05:02.901654 (+ 323us) server_negotiation.cc:407] Connection header received
0929 19:05:02.901763 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:05:02.901764 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:05:02.901777 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:05:02.901807 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:05:02.902206 (+ 399us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52218: BlockingRecv error: recv got EOF from 127.0.0.1:52218 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260929 19:05:02.912949 22753 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:05:02.911787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52228 (local address 127.0.98.61:43091)
0929 19:05:02.911835 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0929 19:05:02.911838 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 19:05:02.912121 (+ 283us) server_negotiation.cc:407] Connection header received
0929 19:05:02.912387 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:05:02.912388 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:05:02.912400 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:05:02.912434 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:05:02.912878 (+ 444us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52228: BlockingRecv error: recv got EOF from 127.0.0.1:52228 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260929 19:05:02.931434 22753 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:05:02.930422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52242 (local address 127.0.98.61:43091)
0929 19:05:02.930469 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0929 19:05:02.930471 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:05:02.930797 (+ 326us) server_negotiation.cc:407] Connection header received
0929 19:05:02.930910 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:05:02.930912 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:05:02.930924 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:05:02.930956 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:05:02.931370 (+ 414us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52242: BlockingRecv error: recv got EOF from 127.0.0.1:52242 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260929 19:05:02.965610 22753 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:05:02.964870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52256 (local address 127.0.98.61:43091)
0929 19:05:02.964963 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0929 19:05:02.964966 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 19:05:02.965030 (+ 64us) server_negotiation.cc:407] Connection header received
0929 19:05:02.965112 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:05:02.965112 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:05:02.965124 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:05:02.965183 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:05:02.965533 (+ 350us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52256: BlockingRecv error: recv got EOF from 127.0.0.1:52256 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260929 19:05:03.031744 22753 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:05:03.031034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52266 (local address 127.0.98.61:43091)
0929 19:05:03.031093 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0929 19:05:03.031095 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:05:03.031203 (+ 108us) server_negotiation.cc:407] Connection header received
0929 19:05:03.031250 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:05:03.031251 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:05:03.031261 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:05:03.031325 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:05:03.031662 (+ 337us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52266: BlockingRecv error: recv got EOF from 127.0.0.1:52266 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
W20260929 19:05:03.161779 22753 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:05:03.160917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52276 (local address 127.0.98.61:43091)
0929 19:05:03.160982 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0929 19:05:03.160984 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:05:03.161226 (+ 242us) server_negotiation.cc:407] Connection header received
0929 19:05:03.161261 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:05:03.161262 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:05:03.161273 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:05:03.161335 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:05:03.161723 (+ 388us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52276: BlockingRecv error: recv got EOF from 127.0.0.1:52276 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
W20260929 19:05:03.419829 22753 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:05:03.419105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52280 (local address 127.0.98.61:43091)
0929 19:05:03.419174 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0929 19:05:03.419176 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:05:03.419209 (+ 33us) server_negotiation.cc:407] Connection header received
0929 19:05:03.419307 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:05:03.419308 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:05:03.419320 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:05:03.419394 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:05:03.419760 (+ 366us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52280: BlockingRecv error: recv got EOF from 127.0.0.1:52280 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
I20260929 19:05:03.739136 22841 catalog_manager.cc:1259] Loaded cluster ID: ffc692d784e1458080711a14cefda39a
I20260929 19:05:03.739218 22841 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 42a00da358fb43e5af7d11184611c861: loading cluster ID for follower catalog manager: success
I20260929 19:05:03.740058 22841 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 42a00da358fb43e5af7d11184611c861: acquiring CA information for follower catalog manager: success
I20260929 19:05:03.740371 22841 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 42a00da358fb43e5af7d11184611c861: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 19:05:03.808873 22884 catalog_manager.cc:1259] Loaded cluster ID: ffc692d784e1458080711a14cefda39a
I20260929 19:05:03.808950 22884 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d033f97a05654c89a2c695a4d3f638a7: loading cluster ID for follower catalog manager: success
I20260929 19:05:03.809710 22884 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d033f97a05654c89a2c695a4d3f638a7: acquiring CA information for follower catalog manager: success
I20260929 19:05:03.809959 22884 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d033f97a05654c89a2c695a4d3f638a7: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 19:05:03.935566 22954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:03.932895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52290 (local address 127.0.98.61:43091)
0929 19:05:03.933040 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0929 19:05:03.933042 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:05:03.933123 (+ 81us) server_negotiation.cc:407] Connection header received
0929 19:05:03.933222 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:05:03.933223 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:05:03.933282 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:05:03.933365 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 19:05:03.933521 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:03.934019 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:03.934779 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:03.934979 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:03.935053 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:05:03.935229 (+ 176us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 19:05:03.935339 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 19:05:03.935425 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 19:05:03.935449 (+ 24us) server_negotiation.cc:299] Negotiation successful
0929 19:05:03.935468 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":60,"threads_started":1}
I20260929 19:05:03.939661 22961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:03.937389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35304 (local address 127.0.98.60:37183)
0929 19:05:03.937493 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0929 19:05:03.937495 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:05:03.937559 (+ 64us) server_negotiation.cc:407] Connection header received
0929 19:05:03.937623 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:05:03.937624 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:05:03.937657 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:05:03.937724 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 19:05:03.937874 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:03.938346 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:03.939026 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:03.939208 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:03.939258 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:05:03.939371 (+ 113us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 19:05:03.939462 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 19:05:03.939544 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 19:05:03.939585 (+ 41us) server_negotiation.cc:299] Negotiation successful
0929 19:05:03.939601 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82,"thread_start_us":59,"threads_started":1}
I20260929 19:05:03.944058 22954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:03.941220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52296 (local address 127.0.98.61:43091)
0929 19:05:03.941260 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0929 19:05:03.941262 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:05:03.941516 (+ 254us) server_negotiation.cc:407] Connection header received
0929 19:05:03.941582 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:05:03.941583 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:05:03.941595 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:05:03.941645 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 19:05:03.941803 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:03.942298 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:03.943299 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:03.943603 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:03.943654 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:05:03.943804 (+ 150us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 19:05:03.943834 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 19:05:03.943885 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0929 19:05:03.943952 (+ 67us) server_negotiation.cc:299] Negotiation successful
0929 19:05:03.943973 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260929 19:05:03.944226 22968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:03.941289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60766 (local address 127.0.98.62:36031)
0929 19:05:03.941416 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0929 19:05:03.941418 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:05:03.941477 (+ 59us) server_negotiation.cc:407] Connection header received
0929 19:05:03.941569 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:05:03.941570 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:05:03.941621 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:05:03.941699 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 19:05:03.941868 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:03.942612 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:03.943328 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:03.943548 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:03.943632 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:05:03.943808 (+ 176us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 19:05:03.943918 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 19:05:03.943972 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0929 19:05:03.944109 (+ 137us) server_negotiation.cc:299] Negotiation successful
0929 19:05:03.944136 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":62,"threads_started":1}
I20260929 19:05:03.944396 22961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:03.941418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35320 (local address 127.0.98.60:37183)
0929 19:05:03.941448 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0929 19:05:03.941450 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:05:03.941609 (+ 159us) server_negotiation.cc:407] Connection header received
0929 19:05:03.941665 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:05:03.941666 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:05:03.941676 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:05:03.941727 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 19:05:03.941851 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:03.942558 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:03.943581 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:03.943868 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:03.943922 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:05:03.944087 (+ 165us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 19:05:03.944116 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 19:05:03.944162 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0929 19:05:03.944297 (+ 135us) server_negotiation.cc:299] Negotiation successful
0929 19:05:03.944320 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260929 19:05:03.945806 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUbnLHE/build/release/bin/kudu with pid 22606
I20260929 19:05:03.951086 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUbnLHE/build/release/bin/kudu with pid 22681
I20260929 19:05:03.955785 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUbnLHE/build/release/bin/kudu with pid 22756
2026-09-29T19:05:03Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2856 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708447806542-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 19:05:04 dist-test-slave-lqkv krb5kdc[22972](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 19:05:04 dist-test-slave-lqkv krb5kdc[22972](info): set up 2 sockets
Sep 29 19:05:04 dist-test-slave-lqkv krb5kdc[22972](info): commencing operation
krb5kdc: starting...
W20260929 19:05:05.001197 392 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708447806542-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708447806542-392-0/krb5kdc/test-user.keytab.
Sep 29 19:05:05 dist-test-slave-lqkv krb5kdc[22972](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708705, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T19:05:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T19:05:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708447806542-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708447806542-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260929 19:05:05.117657 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUbnLHE/build/release/bin/kudu
/tmp/dist-test-taskUbnLHE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708447806542-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708447806542-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708447806542-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708447806542-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42891
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708447806542-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42891,127.0.98.61:36095,127.0.98.60:42913
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708447806542-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708447806542-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708447806542-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 19:05:05.198683 22985 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 19:05:05.198907 22985 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:05:05.198923 22985 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:05:05.198951 22985 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:05:05.200124 22985 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 19:05:05.200172 22985 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:05:05.200186 22985 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:05:05.200199 22985 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 19:05:05.200212 22985 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 19:05:05.201726 22985 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708447806542-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708447806542-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42891,127.0.98.61:36095,127.0.98.60:42913
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708447806542-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42891
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708447806542-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22985
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708447806542-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 18:09:13 UTC on ccycloud.quasar-hpkalf.root.comops.site
build id 4680913
I20260929 19:05:05.201987 22985 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:05:05.202239 22985 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:05:05.207221 22991 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:05:05.207291 22985 server_base.cc:1034] running on GCE node
W20260929 19:05:05.207221 22993 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:05:05.207260 22990 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:05:05.207739 22985 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:05:05.208105 22985 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:05:05.209276 22985 hybrid_clock.cc:648] HybridClock initialized: now 1790708705209256 us; error 34 us; skew 500 ppm
Sep 29 19:05:05 dist-test-slave-lqkv krb5kdc[22972](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708705, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:05:05.211752 22985 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260929 19:05:05.212252 22985 webserver.cc:460] Webserver started at http://127.0.98.62:40849/ using document root <none> and password file <none>
I20260929 19:05:05.212481 22985 fs_manager.cc:359] Metadata directory not provided
I20260929 19:05:05.212553 22985 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:05:05.212709 22985 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:05:05.213841 22985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708447806542-392-0/minicluster-data/master-0/data/instance:
uuid: "2399ba6486b74bbe9169444f6c9163ea"
format_stamp: "Formatted at 2026-09-29 19:05:05 on dist-test-slave-lqkv"
server_key: "b1f970ed9046017b3881af7e8b5ba99a"
server_key_iv: "c56d23feb57e51bb86543e900a4031c7"
server_key_version: "encryptionkey@0"
I20260929 19:05:05.214221 22985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708447806542-392-0/minicluster-data/master-0/wal/instance:
uuid: "2399ba6486b74bbe9169444f6c9163ea"
format_stamp: "Formatted at 2026-09-29 19:05:05 on dist-test-slave-lqkv"
server_key: "b1f970ed9046017b3881af7e8b5ba99a"
server_key_iv: "c56d23feb57e51bb86543e900a4031c7"
server_key_version: "encryptionkey@0"
I20260929 19:05:05.215552 22985 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260929 19:05:05.216261 23000 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:05:05.216423 22985 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260929 19:05:05.216521 22985 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708447806542-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708447806542-392-0/minicluster-data/master-0/wal
uuid: "2399ba6486b74bbe9169444f6c9163ea"
format_stamp: "Formatted at 2026-09-29 19:05:05 on dist-test-slave-lqkv"
server_key: "b1f970ed9046017b3881af7e8b5ba99a"
server_key_iv: "c56d23feb57e51bb86543e900a4031c7"
server_key_version: "encryptionkey@0"
I20260929 19:05:05.216599 22985 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708447806542-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708447806542-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708447806542-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:05:05.262802 22985 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:05:05.264909 22985 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:05:05.265100 22985 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:05:05.268857 22985 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42891
I20260929 19:05:05.268891 23052 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42891 every 8 connection(s)
I20260929 19:05:05.269264 22985 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708447806542-392-0/minicluster-data/master-0/data/info.pb
I20260929 19:05:05.270260 23053 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:05:05.271826 23053 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42891 } has no permanent_uuid. Determining permanent_uuid...
I20260929 19:05:05.273074 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUbnLHE/build/release/bin/kudu as pid 22985
I20260929 19:05:05.273146 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708447806542-392-0/minicluster-data/master-0/wal/instance
I20260929 19:05:05.273370 392 external_mini_cluster.cc:1428] Setting key 9bd35ac7ba6c2b5112ab8554a17183b0
Sep 29 19:05:05 dist-test-slave-lqkv krb5kdc[22972](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708705, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
I20260929 19:05:05.282541 23056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:05.272641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36182 (local address 127.0.98.62:42891)
0929 19:05:05.272771 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0929 19:05:05.272774 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 19:05:05.272835 (+ 61us) server_negotiation.cc:407] Connection header received
0929 19:05:05.273408 (+ 573us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:05:05.273411 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:05:05.273445 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:05:05.273587 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:05:05.274093 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:05.275272 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:05.276223 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:05.276567 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:05.276647 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:05:05.278939 (+ 2292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:05:05.278947 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:05:05.278947 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:05:05.278955 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:05:05.280889 (+ 1934us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:05:05.281250 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:05:05.281253 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:05:05.281254 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:05:05.281362 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:05:05.281610 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:05:05.281612 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:05:05.281612 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:05:05.281912 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:05:05.282002 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0929 19:05:05.282177 (+ 175us) server_negotiation.cc:299] Negotiation successful
0929 19:05:05.282379 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":103,"server-negotiator.queue_time_us":95,"thread_start_us":62,"threads_started":1}
I20260929 19:05:05.282552 23055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:05.272490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42891 (local address 127.0.0.1:36182)
0929 19:05:05.272648 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0929 19:05:05.272685 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 19:05:05.273247 (+ 562us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:05:05.273595 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:05:05.273597 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:05:05.273841 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:05:05.273993 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:05:05.273997 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:05:05.275365 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:05:05.275366 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:05:05.276129 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:05:05.276132 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:05:05.276666 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:05:05.276668 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:05:05.276760 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:05:05.277314 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:05:05.277324 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:05:05.278853 (+ 1529us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:05:05.280966 (+ 2113us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:05:05.280972 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:05:05.280973 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:05:05.281167 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:05:05.281450 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:05:05.281451 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:05:05.281451 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:05:05.281543 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:05:05.281984 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:05:05.281988 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:05:05.282044 (+ 56us) client_negotiation.cc:769] Sending connection context
0929 19:05:05.282116 (+ 72us) client_negotiation.cc:240] Negotiation successful
0929 19:05:05.282293 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":68,"threads_started":1}
I20260929 19:05:05.283334 23053 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36095 } has no permanent_uuid. Determining permanent_uuid...
I20260929 19:05:05.283676 23055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:05.283564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36095 (local address 127.0.0.1:58540)
0929 19:05:05.283612 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0929 19:05:05.283657 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260929 19:05:05.283792 23003 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36095: connect: Connection refused (error 111)
I20260929 19:05:05.284238 23055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:05.284143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36095 (local address 127.0.0.1:58546)
0929 19:05:05.284190 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0929 19:05:05.284213 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260929 19:05:05.284332 23053 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:36095: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36095: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708447806542-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708447806542-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260929 19:05:05.297961 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUbnLHE/build/release/bin/kudu
/tmp/dist-test-taskUbnLHE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708447806542-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708447806542-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708447806542-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708447806542-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:36095
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708447806542-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42891,127.0.98.61:36095,127.0.98.60:42913
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708447806542-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708447806542-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708447806542-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 19:05:05.333559 23053 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36095 } attempt: 1
I20260929 19:05:05.334076 23055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:05.333934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36095 (local address 127.0.0.1:58560)
0929 19:05:05.334002 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0929 19:05:05.334045 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260929 19:05:05.334424 23055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:05.334303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36095 (local address 127.0.0.1:58568)
0929 19:05:05.334349 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0929 19:05:05.334373 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260929 19:05:05.334512 23053 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:36095: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36095: connect: Connection refused (error 111)
W20260929 19:05:05.378253 23060 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 19:05:05.378528 23060 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:05:05.378573 23060 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:05:05.378614 23060 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:05:05.379781 23060 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 19:05:05.379849 23060 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:05:05.379873 23060 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:05:05.379904 23060 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 19:05:05.379933 23060 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 19:05:05.381495 23060 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708447806542-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708447806542-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42891,127.0.98.61:36095,127.0.98.60:42913
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708447806542-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:36095
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708447806542-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23060
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708447806542-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 18:09:13 UTC on ccycloud.quasar-hpkalf.root.comops.site
build id 4680913
I20260929 19:05:05.381795 23060 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:05:05.382028 23060 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 19:05:05.386945 23066 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:05:05.386955 23068 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:05:05.386955 23065 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:05:05.387187 23060 server_base.cc:1034] running on GCE node
I20260929 19:05:05.387413 23060 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:05:05.387708 23060 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:05:05.388867 23060 hybrid_clock.cc:648] HybridClock initialized: now 1790708705388848 us; error 35 us; skew 500 ppm
Sep 29 19:05:05 dist-test-slave-lqkv krb5kdc[22972](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708705, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:05:05.391287 23060 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260929 19:05:05.391728 23060 webserver.cc:460] Webserver started at http://127.0.98.61:39175/ using document root <none> and password file <none>
I20260929 19:05:05.391966 23060 fs_manager.cc:359] Metadata directory not provided
I20260929 19:05:05.392042 23060 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:05:05.392222 23060 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:05:05.393344 23060 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708447806542-392-0/minicluster-data/master-1/data/instance:
uuid: "06a3d3e221a14b47b1f537f4dd6e1bf7"
format_stamp: "Formatted at 2026-09-29 19:05:05 on dist-test-slave-lqkv"
server_key: "5fe6ab7d8a0eaee9026811e85e47b482"
server_key_iv: "f47811897d5efa7e193dbcf8105385d0"
server_key_version: "encryptionkey@0"
I20260929 19:05:05.393708 23060 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708447806542-392-0/minicluster-data/master-1/wal/instance:
uuid: "06a3d3e221a14b47b1f537f4dd6e1bf7"
format_stamp: "Formatted at 2026-09-29 19:05:05 on dist-test-slave-lqkv"
server_key: "5fe6ab7d8a0eaee9026811e85e47b482"
server_key_iv: "f47811897d5efa7e193dbcf8105385d0"
server_key_version: "encryptionkey@0"
I20260929 19:05:05.395044 23060 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260929 19:05:05.395838 23075 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:05:05.396021 23060 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 19:05:05.396122 23060 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708447806542-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708447806542-392-0/minicluster-data/master-1/wal
uuid: "06a3d3e221a14b47b1f537f4dd6e1bf7"
format_stamp: "Formatted at 2026-09-29 19:05:05 on dist-test-slave-lqkv"
server_key: "5fe6ab7d8a0eaee9026811e85e47b482"
server_key_iv: "f47811897d5efa7e193dbcf8105385d0"
server_key_version: "encryptionkey@0"
I20260929 19:05:05.396209 23060 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708447806542-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708447806542-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708447806542-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:05:05.402644 23053 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36095 } attempt: 2
I20260929 19:05:05.403532 23055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:05.402937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36095 (local address 127.0.0.1:58574)
0929 19:05:05.403407 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0929 19:05:05.403461 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":405}
I20260929 19:05:05.403842 23055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:05.403779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36095 (local address 127.0.0.1:58576)
0929 19:05:05.403802 (+ 23us) negotiation.cc:107] Waiting for socket to connect
0929 19:05:05.403827 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":12}
W20260929 19:05:05.403942 23053 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:36095: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36095: connect: Connection refused (error 111)
I20260929 19:05:05.417284 23060 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:05:05.419560 23060 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:05:05.419728 23060 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:05:05.423902 23060 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:36095
I20260929 19:05:05.423972 23127 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:36095 every 8 connection(s)
I20260929 19:05:05.424328 23060 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708447806542-392-0/minicluster-data/master-1/data/info.pb
I20260929 19:05:05.425403 23128 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:05:05.427034 23128 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42891 } has no permanent_uuid. Determining permanent_uuid...
I20260929 19:05:05.432875 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUbnLHE/build/release/bin/kudu as pid 23060
I20260929 19:05:05.432976 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708447806542-392-0/minicluster-data/master-1/wal/instance
I20260929 19:05:05.433203 392 external_mini_cluster.cc:1428] Setting key 75cc8157a02484c328423bc2746d9ea8
Sep 29 19:05:05 dist-test-slave-lqkv krb5kdc[22972](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708705, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 19:05:05.437163 23056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:05.427870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36186 (local address 127.0.98.62:42891)
0929 19:05:05.427952 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0929 19:05:05.427954 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:05:05.428141 (+ 187us) server_negotiation.cc:407] Connection header received
0929 19:05:05.428911 (+ 770us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:05:05.428912 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:05:05.428925 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:05:05.428972 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:05:05.429603 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:05.430088 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:05.431104 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:05.431313 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:05.431425 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:05:05.433985 (+ 2560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:05:05.433987 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:05:05.433987 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:05:05.433995 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:05:05.435483 (+ 1488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:05:05.435999 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:05:05.435999 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:05:05.436000 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:05:05.436053 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:05:05.436368 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:05:05.436370 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:05:05.436370 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:05:05.436525 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:05:05.436590 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0929 19:05:05.437013 (+ 423us) server_negotiation.cc:299] Negotiation successful
0929 19:05:05.437094 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 19:05:05.437419 23130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:05.427800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42891 (local address 127.0.0.1:36186)
0929 19:05:05.427971 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0929 19:05:05.428031 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0929 19:05:05.428756 (+ 725us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:05:05.429052 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:05:05.429055 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:05:05.429303 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:05:05.429528 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:05:05.429532 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:05:05.430187 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:05:05.430188 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:05:05.430994 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:05:05.430996 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:05:05.431452 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:05:05.431454 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:05:05.431582 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:05:05.432065 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:05:05.432073 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:05:05.433869 (+ 1796us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:05:05.435582 (+ 1713us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:05:05.435588 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:05:05.435591 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:05:05.435923 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:05:05.436117 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:05:05.436118 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:05:05.436119 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:05:05.436275 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:05:05.436606 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:05:05.436613 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:05:05.436831 (+ 218us) client_negotiation.cc:769] Sending connection context
0929 19:05:05.436962 (+ 131us) client_negotiation.cc:240] Negotiation successful
0929 19:05:05.437160 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":63,"threads_started":1}
I20260929 19:05:05.438028 23128 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36095 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Sep 29 19:05:05 dist-test-slave-lqkv krb5kdc[22972](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708705, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260929 19:05:05.448040 23130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:05.438325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36095 (local address 127.0.0.1:58588)
0929 19:05:05.438382 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0929 19:05:05.438395 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 19:05:05.438454 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:05:05.438653 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:05:05.438654 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:05:05.438830 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:05:05.438899 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:05:05.438901 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:05:05.439764 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:05:05.439766 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:05:05.440628 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:05:05.440631 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:05:05.441154 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:05:05.441155 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:05:05.441239 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:05:05.441634 (+ 395us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:05:05.441640 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:05:05.444841 (+ 3201us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:05:05.446802 (+ 1961us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:05:05.446803 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:05:05.446804 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:05:05.447003 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:05:05.447236 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:05:05.447237 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:05:05.447237 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:05:05.447293 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:05:05.447675 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:05:05.447676 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:05:05.447745 (+ 69us) client_negotiation.cc:769] Sending connection context
0929 19:05:05.447811 (+ 66us) client_negotiation.cc:240] Negotiation successful
0929 19:05:05.447921 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260929 19:05:05.448136 23132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:05.438364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58588 (local address 127.0.98.61:36095)
0929 19:05:05.438516 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0929 19:05:05.438521 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 19:05:05.438542 (+ 21us) server_negotiation.cc:407] Connection header received
0929 19:05:05.438561 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:05:05.438562 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:05:05.438573 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:05:05.438633 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:05:05.438977 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:05.439683 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:05.440713 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:05.441074 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:05.441133 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:05:05.444945 (+ 3812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:05:05.444959 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:05:05.444959 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:05:05.444965 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:05:05.446712 (+ 1747us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:05:05.447113 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:05:05.447119 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:05:05.447120 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:05:05.447171 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:05:05.447400 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:05:05.447401 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:05:05.447401 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:05:05.447587 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:05:05.447725 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0929 19:05:05.447846 (+ 121us) server_negotiation.cc:299] Negotiation successful
0929 19:05:05.447998 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":63,"server-negotiator.queue_time_us":119,"thread_start_us":91,"threads_started":1}
I20260929 19:05:05.448690 23128 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42913 } has no permanent_uuid. Determining permanent_uuid...
I20260929 19:05:05.449087 23130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:05.448956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42913 (local address 127.0.0.1:52926)
0929 19:05:05.449009 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0929 19:05:05.449068 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260929 19:05:05.449246 23079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42913: connect: Connection refused (error 111)
I20260929 19:05:05.449680 23130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:05.449590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42913 (local address 127.0.0.1:52938)
0929 19:05:05.449636 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0929 19:05:05.449659 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260929 19:05:05.449805 23128 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:42913: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42913: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708447806542-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708447806542-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260929 19:05:05.460366 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUbnLHE/build/release/bin/kudu
/tmp/dist-test-taskUbnLHE/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708447806542-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708447806542-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708447806542-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708447806542-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:42913
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708447806542-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42891,127.0.98.61:36095,127.0.98.60:42913
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708447806542-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708447806542-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708447806542-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 19:05:05.495115 23053 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36095 } attempt: 3
I20260929 19:05:05.499038 23128 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42913 } attempt: 1
I20260929 19:05:05.499416 23130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:05.499291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42913 (local address 127.0.0.1:52942)
0929 19:05:05.499345 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0929 19:05:05.499376 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
Sep 29 19:05:05 dist-test-slave-lqkv krb5kdc[22972](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708705, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260929 19:05:05.499939 23130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:05.499809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42913 (local address 127.0.0.1:52954)
0929 19:05:05.499871 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0929 19:05:05.499910 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260929 19:05:05.500101 23128 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:42913: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42913: connect: Connection refused (error 111)
I20260929 19:05:05.502810 23132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:05.495624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58596 (local address 127.0.98.61:36095)
0929 19:05:05.495705 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0929 19:05:05.495707 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:05:05.495717 (+ 10us) server_negotiation.cc:407] Connection header received
0929 19:05:05.495746 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:05:05.495747 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:05:05.495759 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:05:05.495831 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:05:05.496185 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:05.496710 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:05.497483 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:05.497701 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:05.497786 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:05:05.500101 (+ 2315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:05:05.500102 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:05:05.500103 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:05:05.500108 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:05:05.501668 (+ 1560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:05:05.502032 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:05:05.502034 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:05:05.502034 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:05:05.502091 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:05:05.502293 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:05:05.502294 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:05:05.502295 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:05:05.502415 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:05:05.502487 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 19:05:05.502647 (+ 160us) server_negotiation.cc:299] Negotiation successful
0929 19:05:05.502722 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 19:05:05.502816 23055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:05.495542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36095 (local address 127.0.0.1:58596)
0929 19:05:05.495595 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0929 19:05:05.495609 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 19:05:05.495679 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:05:05.495839 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:05:05.495839 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:05:05.496011 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:05:05.496081 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:05:05.496082 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:05:05.496799 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:05:05.496852 (+ 53us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:05:05.497386 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:05:05.497387 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:05:05.497784 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:05:05.497785 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:05:05.497866 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:05:05.498344 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:05:05.498349 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:05:05.499891 (+ 1542us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:05:05.501771 (+ 1880us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:05:05.501772 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:05:05.501773 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:05:05.501948 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:05:05.502164 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:05:05.502165 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:05:05.502165 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:05:05.502222 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:05:05.502491 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:05:05.502492 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:05:05.502538 (+ 46us) client_negotiation.cc:769] Sending connection context
0929 19:05:05.502622 (+ 84us) client_negotiation.cc:240] Negotiation successful
0929 19:05:05.502722 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260929 19:05:05.503367 23053 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42913 } has no permanent_uuid. Determining permanent_uuid...
I20260929 19:05:05.503695 23055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:05.503593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42913 (local address 127.0.0.1:52968)
0929 19:05:05.503640 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0929 19:05:05.503667 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260929 19:05:05.504117 23055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:05.504017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42913 (local address 127.0.0.1:52974)
0929 19:05:05.504068 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0929 19:05:05.504097 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260929 19:05:05.504215 23053 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:42913: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42913: connect: Connection refused (error 111)
I20260929 19:05:05.535382 23053 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42913 } attempt: 1
I20260929 19:05:05.535852 23055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:05.535722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42913 (local address 127.0.0.1:52990)
0929 19:05:05.535783 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0929 19:05:05.535822 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260929 19:05:05.536202 23055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:05.536105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42913 (local address 127.0.0.1:53000)
0929 19:05:05.536151 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0929 19:05:05.536179 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260929 19:05:05.536288 23053 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:42913: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42913: connect: Connection refused (error 111)
W20260929 19:05:05.542678 23135 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 19:05:05.542936 23135 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 19:05:05.542973 23135 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 19:05:05.543016 23135 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 19:05:05.544183 23135 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 19:05:05.544248 23135 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 19:05:05.544272 23135 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 19:05:05.544296 23135 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 19:05:05.544322 23135 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 19:05:05.545924 23135 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708447806542-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708447806542-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42891,127.0.98.61:36095,127.0.98.60:42913
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708447806542-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:42913
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708447806542-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23135
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708447806542-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 29 Sep 2026 18:09:13 UTC on ccycloud.quasar-hpkalf.root.comops.site
build id 4680913
I20260929 19:05:05.546201 23135 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 19:05:05.546435 23135 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 19:05:05.551592 23135 server_base.cc:1034] running on GCE node
W20260929 19:05:05.551558 23141 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:05:05.551573 23143 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 19:05:05.551566 23140 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 19:05:05.552054 23135 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 19:05:05.552379 23135 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 19:05:05.553555 23135 hybrid_clock.cc:648] HybridClock initialized: now 1790708705553540 us; error 28 us; skew 500 ppm
Sep 29 19:05:05 dist-test-slave-lqkv krb5kdc[22972](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708705, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 19:05:05.556149 23135 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260929 19:05:05.556620 23135 webserver.cc:460] Webserver started at http://127.0.98.60:42925/ using document root <none> and password file <none>
I20260929 19:05:05.556908 23135 fs_manager.cc:359] Metadata directory not provided
I20260929 19:05:05.556982 23135 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 19:05:05.557144 23135 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 19:05:05.558208 23135 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708447806542-392-0/minicluster-data/master-2/data/instance:
uuid: "01cdb18cc48a4c0ab9d8b982248deef6"
format_stamp: "Formatted at 2026-09-29 19:05:05 on dist-test-slave-lqkv"
server_key: "620bd965ebc777f935b2dfd9de0e6caf"
server_key_iv: "99f6416fbed7fd3d84de58ea14ab5f95"
server_key_version: "encryptionkey@0"
I20260929 19:05:05.558585 23135 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708447806542-392-0/minicluster-data/master-2/wal/instance:
uuid: "01cdb18cc48a4c0ab9d8b982248deef6"
format_stamp: "Formatted at 2026-09-29 19:05:05 on dist-test-slave-lqkv"
server_key: "620bd965ebc777f935b2dfd9de0e6caf"
server_key_iv: "99f6416fbed7fd3d84de58ea14ab5f95"
server_key_version: "encryptionkey@0"
I20260929 19:05:05.559975 23135 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260929 19:05:05.560926 23150 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 19:05:05.561102 23135 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 19:05:05.561201 23135 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708447806542-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708447806542-392-0/minicluster-data/master-2/wal
uuid: "01cdb18cc48a4c0ab9d8b982248deef6"
format_stamp: "Formatted at 2026-09-29 19:05:05 on dist-test-slave-lqkv"
server_key: "620bd965ebc777f935b2dfd9de0e6caf"
server_key_iv: "99f6416fbed7fd3d84de58ea14ab5f95"
server_key_version: "encryptionkey@0"
I20260929 19:05:05.561290 23135 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708447806542-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708447806542-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708447806542-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 19:05:05.568277 23128 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42913 } attempt: 2
I20260929 19:05:05.568841 23130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:05.568675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42913 (local address 127.0.0.1:53014)
0929 19:05:05.568736 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0929 19:05:05.568782 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260929 19:05:05.569245 23130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:05.569124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42913 (local address 127.0.0.1:53016)
0929 19:05:05.569193 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0929 19:05:05.569224 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260929 19:05:05.569371 23128 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:42913: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42913: connect: Connection refused (error 111)
I20260929 19:05:05.583065 23135 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 19:05:05.585593 23135 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 19:05:05.585754 23135 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 19:05:05.589790 23135 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:42913
I20260929 19:05:05.589843 23202 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:42913 every 8 connection(s)
I20260929 19:05:05.590219 23135 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708447806542-392-0/minicluster-data/master-2/data/info.pb
I20260929 19:05:05.591225 23203 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 19:05:05.592767 23203 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42891 } has no permanent_uuid. Determining permanent_uuid...
I20260929 19:05:05.595523 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUbnLHE/build/release/bin/kudu as pid 23135
I20260929 19:05:05.595650 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUbnLHE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790708447806542-392-0/minicluster-data/master-2/wal/instance
I20260929 19:05:05.595858 392 external_mini_cluster.cc:1428] Setting key 4821f34fc1ed5dd31f98f5f3f4244685
Sep 29 19:05:05 dist-test-slave-lqkv krb5kdc[22972](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708705, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 29 19:05:05 dist-test-slave-lqkv krb5kdc[22972](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708705, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260929 19:05:05.602713 23056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:05.593648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36196 (local address 127.0.98.62:42891)
0929 19:05:05.593718 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0929 19:05:05.593719 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 19:05:05.593928 (+ 209us) server_negotiation.cc:407] Connection header received
0929 19:05:05.594437 (+ 509us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:05:05.594438 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:05:05.594450 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:05:05.594507 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:05:05.595133 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:05.595607 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:05.597037 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:05.597226 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:05.597304 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:05:05.599488 (+ 2184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:05:05.599490 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:05:05.599490 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:05:05.599498 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:05:05.601053 (+ 1555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:05:05.601546 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:05:05.601548 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:05:05.601548 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:05:05.601608 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:05:05.601881 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:05:05.601882 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:05:05.601882 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:05:05.602059 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:05:05.602132 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 19:05:05.602494 (+ 362us) server_negotiation.cc:299] Negotiation successful
0929 19:05:05.602601 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 19:05:05.602922 23205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:05.593615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42891 (local address 127.0.0.1:36196)
0929 19:05:05.593808 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0929 19:05:05.593843 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0929 19:05:05.594326 (+ 483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:05:05.594587 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:05:05.594590 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:05:05.594835 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:05:05.595037 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:05:05.595041 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:05:05.595726 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:05:05.595728 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:05:05.596955 (+ 1227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:05:05.596957 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:05:05.597330 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:05:05.597332 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:05:05.597461 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:05:05.597992 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:05:05.598005 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:05:05.599388 (+ 1383us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:05:05.601152 (+ 1764us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:05:05.601159 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:05:05.601161 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:05:05.601453 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:05:05.601677 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:05:05.601677 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:05:05.601678 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:05:05.601816 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:05:05.602141 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:05:05.602147 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:05:05.602330 (+ 183us) client_negotiation.cc:769] Sending connection context
0929 19:05:05.602469 (+ 139us) client_negotiation.cc:240] Negotiation successful
0929 19:05:05.602630 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":104,"threads_started":1}
I20260929 19:05:05.603538 23203 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36095 } has no permanent_uuid. Determining permanent_uuid...
I20260929 19:05:05.604883 23207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:05.596259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53018 (local address 127.0.98.60:42913)
0929 19:05:05.596448 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0929 19:05:05.596453 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 19:05:05.596493 (+ 40us) server_negotiation.cc:407] Connection header received
0929 19:05:05.596550 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:05:05.596552 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:05:05.596571 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:05:05.596653 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:05:05.597305 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:05.597991 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:05.598790 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:05.599243 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:05.599336 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:05:05.601713 (+ 2377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:05:05.601721 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:05:05.601722 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:05:05.601731 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:05:05.603636 (+ 1905us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:05:05.603994 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:05:05.603998 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:05:05.603998 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:05:05.604069 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:05:05.604313 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:05:05.604314 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:05:05.604314 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:05:05.604446 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:05:05.604620 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0929 19:05:05.604670 (+ 50us) server_negotiation.cc:299] Negotiation successful
0929 19:05:05.604748 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":131,"threads_started":1}
Sep 29 19:05:05 dist-test-slave-lqkv krb5kdc[22972](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708705, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 29 19:05:05 dist-test-slave-lqkv krb5kdc[22972](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708705, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260929 19:05:05.611052 23132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:05.603959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58610 (local address 127.0.98.61:36095)
0929 19:05:05.604023 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0929 19:05:05.604025 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:05:05.604072 (+ 47us) server_negotiation.cc:407] Connection header received
0929 19:05:05.604154 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:05:05.604155 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:05:05.604166 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:05:05.604219 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:05:05.604498 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:05.605233 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:05.606148 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:05.606345 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:05.606398 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:05:05.608540 (+ 2142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:05:05.608541 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:05:05.608542 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:05:05.608549 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:05:05.609916 (+ 1367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:05:05.610275 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:05:05.610276 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:05:05.610277 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:05:05.610336 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:05:05.610534 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:05:05.610535 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:05:05.610535 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:05:05.610652 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:05:05.610721 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0929 19:05:05.610880 (+ 159us) server_negotiation.cc:299] Negotiation successful
0929 19:05:05.610954 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260929 19:05:05.611061 23205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:05.603883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36095 (local address 127.0.0.1:58610)
0929 19:05:05.604012 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0929 19:05:05.604037 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 19:05:05.604081 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:05:05.604238 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:05:05.604239 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:05:05.604390 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:05:05.604425 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:05:05.604426 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:05:05.605331 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:05:05.605332 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:05:05.606063 (+ 731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:05:05.606065 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:05:05.606419 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:05:05.606420 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:05:05.606485 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:05:05.606951 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:05:05.606956 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:05:05.608420 (+ 1464us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:05:05.610015 (+ 1595us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:05:05.610016 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:05:05.610017 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:05:05.610200 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:05:05.610403 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:05:05.610404 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:05:05.610404 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:05:05.610461 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:05:05.610723 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:05:05.610724 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:05:05.610770 (+ 46us) client_negotiation.cc:769] Sending connection context
0929 19:05:05.610853 (+ 83us) client_negotiation.cc:240] Negotiation successful
0929 19:05:05.610955 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111}
I20260929 19:05:05.611423 23053 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42913 } attempt: 2
I20260929 19:05:05.611593 23203 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42913 } has no permanent_uuid. Determining permanent_uuid...
I20260929 19:05:05.614521 23056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:05.605719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36210 (local address 127.0.98.62:42891)
0929 19:05:05.605764 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0929 19:05:05.605766 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:05:05.605812 (+ 46us) server_negotiation.cc:407] Connection header received
0929 19:05:05.605881 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:05:05.605882 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:05:05.605889 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:05:05.605926 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:05:05.606466 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:05.607076 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:05.607905 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:05.608220 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:05.608276 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:05:05.610807 (+ 2531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:05:05.610809 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:05:05.610809 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:05:05.610817 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:05:05.613269 (+ 2452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:05:05.613656 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:05:05.613658 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:05:05.613658 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:05:05.613716 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:05:05.613930 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:05:05.613931 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:05:05.613932 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:05:05.614050 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:05:05.614238 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0929 19:05:05.614274 (+ 36us) server_negotiation.cc:299] Negotiation successful
0929 19:05:05.614348 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
Sep 29 19:05:05 dist-test-slave-lqkv krb5kdc[22972](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708705, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 29 19:05:05 dist-test-slave-lqkv krb5kdc[22972](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708705, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 29 19:05:05 dist-test-slave-lqkv krb5kdc[22972](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708705, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260929 19:05:05.621523 23055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:05.611720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42913 (local address 127.0.0.1:53030)
0929 19:05:05.611781 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0929 19:05:05.611791 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 19:05:05.611826 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:05:05.612023 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:05:05.612024 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:05:05.612260 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:05:05.612621 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:05:05.612624 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:05:05.613473 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:05:05.613474 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:05:05.614331 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:05:05.614333 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:05:05.614840 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:05:05.614841 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:05:05.614900 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:05:05.615407 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:05:05.615413 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:05:05.616919 (+ 1506us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:05:05.620185 (+ 3266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:05:05.620186 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:05:05.620187 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:05:05.620418 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:05:05.620679 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:05:05.620680 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:05:05.620680 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:05:05.620743 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:05:05.621149 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:05:05.621150 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:05:05.621203 (+ 53us) client_negotiation.cc:769] Sending connection context
0929 19:05:05.621298 (+ 95us) client_negotiation.cc:240] Negotiation successful
0929 19:05:05.621407 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260929 19:05:05.621573 23207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:05.611744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53030 (local address 127.0.98.60:42913)
0929 19:05:05.611893 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0929 19:05:05.611894 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 19:05:05.611909 (+ 15us) server_negotiation.cc:407] Connection header received
0929 19:05:05.611926 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:05:05.611927 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:05:05.611944 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:05:05.611983 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:05:05.612708 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:05.613395 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:05.614427 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:05.614751 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:05.615022 (+ 271us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:05:05.617023 (+ 2001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:05:05.617025 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:05:05.617025 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:05:05.617033 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:05:05.620087 (+ 3054us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:05:05.620501 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:05:05.620502 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:05:05.620502 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:05:05.620581 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:05:05.620844 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:05:05.620846 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:05:05.620846 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:05:05.621081 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:05:05.621157 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 19:05:05.621342 (+ 185us) server_negotiation.cc:299] Negotiation successful
0929 19:05:05.621472 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132}
I20260929 19:05:05.622112 23205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:05.611856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42913 (local address 127.0.0.1:53040)
0929 19:05:05.612011 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0929 19:05:05.612025 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 19:05:05.612069 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:05:05.612222 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:05:05.612223 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:05:05.612353 (+ 130us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:05:05.612381 (+ 28us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:05:05.612382 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:05:05.613292 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:05:05.613294 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:05:05.614208 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:05:05.614211 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:05:05.614780 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:05:05.614781 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:05:05.614838 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:05:05.615558 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:05:05.615563 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:05:05.617548 (+ 1985us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:05:05.619775 (+ 2227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:05:05.619776 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:05:05.619777 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:05:05.620127 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:05:05.620416 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:05:05.620417 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:05:05.620417 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:05:05.620488 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:05:05.621726 (+ 1238us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:05:05.621727 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:05:05.621775 (+ 48us) client_negotiation.cc:769] Sending connection context
0929 19:05:05.621865 (+ 90us) client_negotiation.cc:240] Negotiation successful
0929 19:05:05.621971 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147}
I20260929 19:05:05.622478 23208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:05.611887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53040 (local address 127.0.98.60:42913)
0929 19:05:05.612115 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0929 19:05:05.612116 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 19:05:05.612122 (+ 6us) server_negotiation.cc:407] Connection header received
0929 19:05:05.612137 (+ 15us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:05:05.612137 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:05:05.612145 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:05:05.612221 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:05:05.612458 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:05.613202 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:05.614307 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:05.614668 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:05.614731 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:05:05.617661 (+ 2930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:05:05.617662 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:05:05.617663 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:05:05.617669 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:05:05.619597 (+ 1928us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:05:05.620205 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:05:05.620206 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:05:05.620206 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:05:05.620312 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:05:05.620906 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:05:05.620907 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:05:05.620907 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:05:05.621080 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:05:05.621151 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 19:05:05.622225 (+ 1074us) server_negotiation.cc:299] Negotiation successful
0929 19:05:05.622301 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":208,"thread_start_us":159,"threads_started":1}
I20260929 19:05:05.623062 23132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:05.615124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58616 (local address 127.0.98.61:36095)
0929 19:05:05.615182 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0929 19:05:05.615184 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:05:05.615192 (+ 8us) server_negotiation.cc:407] Connection header received
0929 19:05:05.615215 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:05:05.615216 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:05:05.615224 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:05:05.615268 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:05:05.615733 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:05.616189 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:05.616998 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:05.617234 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:05.617294 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:05:05.620020 (+ 2726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:05:05.620021 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:05:05.620021 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:05:05.620028 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:05:05.621646 (+ 1618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:05:05.622096 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:05:05.622097 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:05:05.622097 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:05:05.622159 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:05:05.622393 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:05:05.622394 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:05:05.622394 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:05:05.622554 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:05:05.622701 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0929 19:05:05.622834 (+ 133us) server_negotiation.cc:299] Negotiation successful
0929 19:05:05.622924 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260929 19:05:05.623627 23053 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2399ba6486b74bbe9169444f6c9163ea: Bootstrap starting.
I20260929 19:05:05.624325 23053 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2399ba6486b74bbe9169444f6c9163ea: Neither blocks nor log segments found. Creating new log.
I20260929 19:05:05.624305 23203 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 01cdb18cc48a4c0ab9d8b982248deef6: Bootstrap starting.
I20260929 19:05:05.624647 23053 log.cc:824] T 00000000000000000000000000000000 P 2399ba6486b74bbe9169444f6c9163ea: Log is configured to *not* fsync() on all Append() calls
I20260929 19:05:05.624935 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 19:05:05.625169 23203 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 01cdb18cc48a4c0ab9d8b982248deef6: Neither blocks nor log segments found. Creating new log.
I20260929 19:05:05.625500 23203 log.cc:824] T 00000000000000000000000000000000 P 01cdb18cc48a4c0ab9d8b982248deef6: Log is configured to *not* fsync() on all Append() calls
I20260929 19:05:05.625864 23053 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2399ba6486b74bbe9169444f6c9163ea: No bootstrap required, opened a new log
I20260929 19:05:05.626415 23203 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 01cdb18cc48a4c0ab9d8b982248deef6: No bootstrap required, opened a new log
I20260929 19:05:05.625928 23027 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "2399ba6486b74bbe9169444f6c9163ea"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36210
I20260929 19:05:05.627938 23053 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2399ba6486b74bbe9169444f6c9163ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2399ba6486b74bbe9169444f6c9163ea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42891 } } peers { permanent_uuid: "06a3d3e221a14b47b1f537f4dd6e1bf7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36095 } } peers { permanent_uuid: "01cdb18cc48a4c0ab9d8b982248deef6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42913 } }
I20260929 19:05:05.628074 23053 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2399ba6486b74bbe9169444f6c9163ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2399ba6486b74bbe9169444f6c9163ea, State: Initialized, Role: FOLLOWER
I20260929 19:05:05.628163 23053 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2399ba6486b74bbe9169444f6c9163ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2399ba6486b74bbe9169444f6c9163ea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42891 } } peers { permanent_uuid: "06a3d3e221a14b47b1f537f4dd6e1bf7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36095 } } peers { permanent_uuid: "01cdb18cc48a4c0ab9d8b982248deef6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42913 } }
I20260929 19:05:05.628479 23053 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2399ba6486b74bbe9169444f6c9163ea [sys.catalog]: configured and running, proceeding with master startup.
I20260929 19:05:05.628525 23027 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2399ba6486b74bbe9169444f6c9163ea [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 19:05:05.628499 23211 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2399ba6486b74bbe9169444f6c9163ea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2399ba6486b74bbe9169444f6c9163ea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42891 } } peers { permanent_uuid: "06a3d3e221a14b47b1f537f4dd6e1bf7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36095 } } peers { permanent_uuid: "01cdb18cc48a4c0ab9d8b982248deef6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42913 } } }
I20260929 19:05:05.628579 23027 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2399ba6486b74bbe9169444f6c9163ea [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:05:05.628585 23211 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2399ba6486b74bbe9169444f6c9163ea [sys.catalog]: This master's current role is: FOLLOWER
I20260929 19:05:05.628679 23203 raft_consensus.cc:348] T 00000000000000000000000000000000 P 01cdb18cc48a4c0ab9d8b982248deef6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2399ba6486b74bbe9169444f6c9163ea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42891 } } peers { permanent_uuid: "06a3d3e221a14b47b1f537f4dd6e1bf7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36095 } } peers { permanent_uuid: "01cdb18cc48a4c0ab9d8b982248deef6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42913 } }
I20260929 19:05:05.628885 23203 raft_consensus.cc:729] T 00000000000000000000000000000000 P 01cdb18cc48a4c0ab9d8b982248deef6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01cdb18cc48a4c0ab9d8b982248deef6, State: Initialized, Role: FOLLOWER
I20260929 19:05:05.628980 23203 consensus_queue.cc:260] T 00000000000000000000000000000000 P 01cdb18cc48a4c0ab9d8b982248deef6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2399ba6486b74bbe9169444f6c9163ea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42891 } } peers { permanent_uuid: "06a3d3e221a14b47b1f537f4dd6e1bf7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36095 } } peers { permanent_uuid: "01cdb18cc48a4c0ab9d8b982248deef6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42913 } }
I20260929 19:05:05.629292 23203 sys_catalog.cc:564] T 00000000000000000000000000000000 P 01cdb18cc48a4c0ab9d8b982248deef6 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 19:05:05.629256 23216 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01cdb18cc48a4c0ab9d8b982248deef6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2399ba6486b74bbe9169444f6c9163ea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42891 } } peers { permanent_uuid: "06a3d3e221a14b47b1f537f4dd6e1bf7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36095 } } peers { permanent_uuid: "01cdb18cc48a4c0ab9d8b982248deef6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42913 } } }
I20260929 19:05:05.629348 23216 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01cdb18cc48a4c0ab9d8b982248deef6 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 19:05:05.629501 23027 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2399ba6486b74bbe9169444f6c9163ea [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2399ba6486b74bbe9169444f6c9163ea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42891 } } peers { permanent_uuid: "06a3d3e221a14b47b1f537f4dd6e1bf7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36095 } } peers { permanent_uuid: "01cdb18cc48a4c0ab9d8b982248deef6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42913 } }
I20260929 19:05:05.629760 23027 leader_election.cc:288] T 00000000000000000000000000000000 P 2399ba6486b74bbe9169444f6c9163ea [CANDIDATE]: Term 1 election: Requested vote from peers 06a3d3e221a14b47b1f537f4dd6e1bf7 (127.0.98.61:36095), 01cdb18cc48a4c0ab9d8b982248deef6 (127.0.98.60:42913)
I20260929 19:05:05.630605 23177 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2399ba6486b74bbe9169444f6c9163ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "01cdb18cc48a4c0ab9d8b982248deef6"
W20260929 19:05:05.630681 23225 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2399ba6486b74bbe9169444f6c9163ea: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 19:05:05.630730 23225 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 19:05:05.630738 23177 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 01cdb18cc48a4c0ab9d8b982248deef6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:05:05.631794 23177 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 01cdb18cc48a4c0ab9d8b982248deef6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2399ba6486b74bbe9169444f6c9163ea in term 1.
I20260929 19:05:05.632092 23004 leader_election.cc:302] T 00000000000000000000000000000000 P 2399ba6486b74bbe9169444f6c9163ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01cdb18cc48a4c0ab9d8b982248deef6, 2399ba6486b74bbe9169444f6c9163ea; no voters:
I20260929 19:05:05.632211 23211 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2399ba6486b74bbe9169444f6c9163ea [term 1 FOLLOWER]: Leader election won for term 1
I20260929 19:05:05.632351 23211 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2399ba6486b74bbe9169444f6c9163ea [term 1 LEADER]: Becoming Leader. State: Replica: 2399ba6486b74bbe9169444f6c9163ea, State: Running, Role: LEADER
I20260929 19:05:05.632424 23211 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2399ba6486b74bbe9169444f6c9163ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2399ba6486b74bbe9169444f6c9163ea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42891 } } peers { permanent_uuid: "06a3d3e221a14b47b1f537f4dd6e1bf7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36095 } } peers { permanent_uuid: "01cdb18cc48a4c0ab9d8b982248deef6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42913 } }
I20260929 19:05:05.632848 23211 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2399ba6486b74bbe9169444f6c9163ea [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2399ba6486b74bbe9169444f6c9163ea. Latest consensus state: current_term: 1 leader_uuid: "2399ba6486b74bbe9169444f6c9163ea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2399ba6486b74bbe9169444f6c9163ea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42891 } } peers { permanent_uuid: "06a3d3e221a14b47b1f537f4dd6e1bf7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36095 } } peers { permanent_uuid: "01cdb18cc48a4c0ab9d8b982248deef6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42913 } } }
I20260929 19:05:05.632915 23211 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2399ba6486b74bbe9169444f6c9163ea [sys.catalog]: This master's current role is: LEADER
W20260929 19:05:05.634020 23235 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 01cdb18cc48a4c0ab9d8b982248deef6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 19:05:05.634104 23235 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 19:05:05.631120 22985 master_runner.cc:186] Error getting master registration for 127.0.98.61:36095: OK, instance_id { permanent_uuid: "06a3d3e221a14b47b1f537f4dd6e1bf7" instance_seqno: 1790708705422780 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 19:05:05.635592 22985 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 19:05:05.634735 23135 master_runner.cc:186] Error getting master registration for 127.0.98.61:36095: OK, instance_id { permanent_uuid: "06a3d3e221a14b47b1f537f4dd6e1bf7" instance_seqno: 1790708705422780 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 19:05:05.636997 23135 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 19:05:05.640960 23239 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 19:05:05.641171 23239 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 19:05:05.645409 23177 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 01cdb18cc48a4c0ab9d8b982248deef6 [term 1 FOLLOWER]: Refusing update from remote peer 2399ba6486b74bbe9169444f6c9163ea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 19:05:05.645767 23056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:05.634684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36222 (local address 127.0.98.62:42891)
0929 19:05:05.634726 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0929 19:05:05.634728 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:05:05.635778 (+ 1050us) server_negotiation.cc:407] Connection header received
0929 19:05:05.635841 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:05:05.635842 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:05:05.635855 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:05:05.635923 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:05:05.637023 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:05.637760 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:05.639430 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:05.639739 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:05.639801 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:05:05.641466 (+ 1665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:05:05.641467 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:05:05.641468 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:05:05.641476 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:05:05.643624 (+ 2148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:05:05.644606 (+ 982us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:05:05.644607 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:05:05.644607 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:05:05.644658 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:05:05.645146 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:05:05.645147 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:05:05.645148 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:05:05.645298 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:05:05.645370 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 19:05:05.645588 (+ 218us) server_negotiation.cc:299] Negotiation successful
0929 19:05:05.645677 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260929 19:05:05.646174 23211 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2399ba6486b74bbe9169444f6c9163ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "01cdb18cc48a4c0ab9d8b982248deef6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 19:05:05.647257 23208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:05.634787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53048 (local address 127.0.98.60:42913)
0929 19:05:05.634819 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0929 19:05:05.634821 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:05:05.635934 (+ 1113us) server_negotiation.cc:407] Connection header received
0929 19:05:05.637019 (+ 1085us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:05:05.637020 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:05:05.637032 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:05:05.637090 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:05:05.638505 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:05.639305 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:05.640908 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:05.641214 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:05.641262 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:05:05.643474 (+ 2212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:05:05.643487 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:05:05.643487 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:05:05.643494 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:05:05.645236 (+ 1742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:05:05.646225 (+ 989us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:05:05.646226 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:05:05.646226 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:05:05.646293 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:05:05.646631 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:05:05.646632 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:05:05.646633 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:05:05.646772 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:05:05.646845 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 19:05:05.647067 (+ 222us) server_negotiation.cc:299] Negotiation successful
0929 19:05:05.647158 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260929 19:05:05.647866 23216 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01cdb18cc48a4c0ab9d8b982248deef6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2399ba6486b74bbe9169444f6c9163ea. Latest consensus state: current_term: 1 leader_uuid: "2399ba6486b74bbe9169444f6c9163ea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2399ba6486b74bbe9169444f6c9163ea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42891 } } peers { permanent_uuid: "06a3d3e221a14b47b1f537f4dd6e1bf7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36095 } } peers { permanent_uuid: "01cdb18cc48a4c0ab9d8b982248deef6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42913 } } }
I20260929 19:05:05.647975 23216 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01cdb18cc48a4c0ab9d8b982248deef6 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 19:05:05.649504 23216 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01cdb18cc48a4c0ab9d8b982248deef6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2399ba6486b74bbe9169444f6c9163ea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2399ba6486b74bbe9169444f6c9163ea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42891 } } peers { permanent_uuid: "06a3d3e221a14b47b1f537f4dd6e1bf7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36095 } } peers { permanent_uuid: "01cdb18cc48a4c0ab9d8b982248deef6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42913 } } }
I20260929 19:05:05.649609 23216 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01cdb18cc48a4c0ab9d8b982248deef6 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 19:05:05.649796 23248 mvcc.cc:205] Tried to move back new op lower bound from 7334742858314592256 to 7334742858271019008. Current Snapshot: MvccSnapshot[applied={T|T < 7334742858314592256}]
I20260929 19:05:05.650132 23239 catalog_manager.cc:1347] Generated new cluster ID: b4cd01921cdb449bbc1fa28b2543fabd
I20260929 19:05:05.650207 23239 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 19:05:05.650310 23132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:05.635288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58618 (local address 127.0.98.61:36095)
0929 19:05:05.635328 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0929 19:05:05.635330 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:05:05.639536 (+ 4206us) server_negotiation.cc:407] Connection header received
0929 19:05:05.639623 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:05:05.639624 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:05:05.639634 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:05:05.639663 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:05:05.640437 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:05.641247 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:05.642285 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:05.642631 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:05.642673 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:05:05.644536 (+ 1863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:05:05.644537 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:05:05.644538 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:05:05.644545 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:05:05.648566 (+ 4021us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:05:05.649107 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:05:05.649108 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:05:05.649108 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:05:05.649189 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:05:05.649620 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:05:05.649621 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:05:05.649622 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:05:05.649770 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:05:05.649857 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0929 19:05:05.650140 (+ 283us) server_negotiation.cc:299] Negotiation successful
0929 19:05:05.650227 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260929 19:05:05.651211 23236 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2399ba6486b74bbe9169444f6c9163ea [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2399ba6486b74bbe9169444f6c9163ea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2399ba6486b74bbe9169444f6c9163ea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42891 } } peers { permanent_uuid: "06a3d3e221a14b47b1f537f4dd6e1bf7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36095 } } peers { permanent_uuid: "01cdb18cc48a4c0ab9d8b982248deef6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42913 } } }
I20260929 19:05:05.651307 23236 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2399ba6486b74bbe9169444f6c9163ea [sys.catalog]: This master's current role is: LEADER
I20260929 19:05:05.651371 23247 mvcc.cc:205] Tried to move back new op lower bound from 7334742858314592256 to 7334742858271019008. Current Snapshot: MvccSnapshot[applied={T|T < 7334742858314592256 or (T in {7334742858314592256})}]
I20260929 19:05:05.660506 23128 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42913 } attempt: 3
Sep 29 19:05:05 dist-test-slave-lqkv krb5kdc[22972](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790708705, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260929 19:05:05.670954 23130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:05.660754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42913 (local address 127.0.0.1:53064)
0929 19:05:05.660854 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0929 19:05:05.660864 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0929 19:05:05.660898 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 19:05:05.661130 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 19:05:05.661131 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 19:05:05.661296 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0929 19:05:05.661340 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:05:05.661341 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:05:05.662198 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:05:05.662199 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:05:05.662963 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 19:05:05.662964 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 19:05:05.663419 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 19:05:05.663420 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 19:05:05.663478 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:05:05.663961 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 19:05:05.663966 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 19:05:05.665820 (+ 1854us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 19:05:05.669634 (+ 3814us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:05:05.669635 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:05:05.669636 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:05:05.669859 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:05:05.670114 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 19:05:05.670115 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 19:05:05.670115 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0929 19:05:05.670178 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 19:05:05.670616 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 19:05:05.670617 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 19:05:05.670668 (+ 51us) client_negotiation.cc:769] Sending connection context
0929 19:05:05.670732 (+ 64us) client_negotiation.cc:240] Negotiation successful
0929 19:05:05.670843 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80}
I20260929 19:05:05.671156 23239 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 19:05:05.671756 23208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:05.660966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53064 (local address 127.0.98.60:42913)
0929 19:05:05.661003 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0929 19:05:05.661004 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 19:05:05.661011 (+ 7us) server_negotiation.cc:407] Connection header received
0929 19:05:05.661032 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:05:05.661032 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:05:05.661041 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:05:05.661080 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:05:05.661406 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:05.662121 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:05.663041 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:05.663322 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:05.663373 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:05:05.666545 (+ 3172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 19:05:05.666547 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 19:05:05.666547 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 19:05:05.666555 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0929 19:05:05.669527 (+ 2972us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:05:05.669961 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:05:05.669962 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:05:05.669962 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:05:05.670039 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 19:05:05.670260 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 19:05:05.670261 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 19:05:05.670262 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 19:05:05.670528 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 19:05:05.670589 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0929 19:05:05.671600 (+ 1011us) server_negotiation.cc:299] Negotiation successful
0929 19:05:05.671688 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260929 19:05:05.671890 23239 catalog_manager.cc:1504] Loading token signing keys...
I20260929 19:05:05.673065 23128 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06a3d3e221a14b47b1f537f4dd6e1bf7: Bootstrap starting.
I20260929 19:05:05.673708 23128 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 06a3d3e221a14b47b1f537f4dd6e1bf7: Neither blocks nor log segments found. Creating new log.
I20260929 19:05:05.674009 23128 log.cc:824] T 00000000000000000000000000000000 P 06a3d3e221a14b47b1f537f4dd6e1bf7: Log is configured to *not* fsync() on all Append() calls
I20260929 19:05:05.674868 23128 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06a3d3e221a14b47b1f537f4dd6e1bf7: No bootstrap required, opened a new log
I20260929 19:05:05.677584 23128 raft_consensus.cc:348] T 00000000000000000000000000000000 P 06a3d3e221a14b47b1f537f4dd6e1bf7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2399ba6486b74bbe9169444f6c9163ea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42891 } } peers { permanent_uuid: "06a3d3e221a14b47b1f537f4dd6e1bf7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36095 } } peers { permanent_uuid: "01cdb18cc48a4c0ab9d8b982248deef6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42913 } }
I20260929 19:05:05.677729 23128 raft_consensus.cc:729] T 00000000000000000000000000000000 P 06a3d3e221a14b47b1f537f4dd6e1bf7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06a3d3e221a14b47b1f537f4dd6e1bf7, State: Initialized, Role: FOLLOWER
I20260929 19:05:05.677815 23128 consensus_queue.cc:260] T 00000000000000000000000000000000 P 06a3d3e221a14b47b1f537f4dd6e1bf7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2399ba6486b74bbe9169444f6c9163ea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42891 } } peers { permanent_uuid: "06a3d3e221a14b47b1f537f4dd6e1bf7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36095 } } peers { permanent_uuid: "01cdb18cc48a4c0ab9d8b982248deef6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42913 } }
I20260929 19:05:05.678177 23128 sys_catalog.cc:564] T 00000000000000000000000000000000 P 06a3d3e221a14b47b1f537f4dd6e1bf7 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 19:05:05.678365 23101 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 06a3d3e221a14b47b1f537f4dd6e1bf7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 19:05:05.679373 23101 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 06a3d3e221a14b47b1f537f4dd6e1bf7 [term 1 FOLLOWER]: Refusing update from remote peer 2399ba6486b74bbe9169444f6c9163ea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 19:05:05.678198 23102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2399ba6486b74bbe9169444f6c9163ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "06a3d3e221a14b47b1f537f4dd6e1bf7"
I20260929 19:05:05.680560 23102 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 06a3d3e221a14b47b1f537f4dd6e1bf7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2399ba6486b74bbe9169444f6c9163ea in term 1.
I20260929 19:05:05.681049 23236 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2399ba6486b74bbe9169444f6c9163ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "06a3d3e221a14b47b1f537f4dd6e1bf7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 19:05:05.683292 23265 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06a3d3e221a14b47b1f537f4dd6e1bf7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2399ba6486b74bbe9169444f6c9163ea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2399ba6486b74bbe9169444f6c9163ea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42891 } } peers { permanent_uuid: "06a3d3e221a14b47b1f537f4dd6e1bf7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36095 } } peers { permanent_uuid: "01cdb18cc48a4c0ab9d8b982248deef6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42913 } } }
I20260929 19:05:05.683435 23265 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06a3d3e221a14b47b1f537f4dd6e1bf7 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 19:05:05.684090 23265 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06a3d3e221a14b47b1f537f4dd6e1bf7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2399ba6486b74bbe9169444f6c9163ea. Latest consensus state: current_term: 1 leader_uuid: "2399ba6486b74bbe9169444f6c9163ea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2399ba6486b74bbe9169444f6c9163ea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42891 } } peers { permanent_uuid: "06a3d3e221a14b47b1f537f4dd6e1bf7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36095 } } peers { permanent_uuid: "01cdb18cc48a4c0ab9d8b982248deef6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42913 } } }
W20260929 19:05:05.684104 23264 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 06a3d3e221a14b47b1f537f4dd6e1bf7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260929 19:05:05.684247 23265 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06a3d3e221a14b47b1f537f4dd6e1bf7 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 19:05:05.684379 23265 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06a3d3e221a14b47b1f537f4dd6e1bf7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2399ba6486b74bbe9169444f6c9163ea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2399ba6486b74bbe9169444f6c9163ea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42891 } } peers { permanent_uuid: "06a3d3e221a14b47b1f537f4dd6e1bf7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36095 } } peers { permanent_uuid: "01cdb18cc48a4c0ab9d8b982248deef6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42913 } } }
I20260929 19:05:05.684487 23265 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06a3d3e221a14b47b1f537f4dd6e1bf7 [sys.catalog]: This master's current role is: FOLLOWER
W20260929 19:05:05.684265 23264 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 19:05:05.697676 23236 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2399ba6486b74bbe9169444f6c9163ea [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2399ba6486b74bbe9169444f6c9163ea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2399ba6486b74bbe9169444f6c9163ea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42891 } } peers { permanent_uuid: "06a3d3e221a14b47b1f537f4dd6e1bf7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36095 } } peers { permanent_uuid: "01cdb18cc48a4c0ab9d8b982248deef6" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42913 } } }
I20260929 19:05:05.697803 23236 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2399ba6486b74bbe9169444f6c9163ea [sys.catalog]: This master's current role is: LEADER
I20260929 19:05:05.699899 23239 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2399ba6486b74bbe9169444f6c9163ea: Generated new TSK 0
I20260929 19:05:05.700109 23239 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 19:05:05.729761 23208 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:05:05.727817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53074 (local address 127.0.98.60:42913)
0929 19:05:05.727855 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0929 19:05:05.727856 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 19:05:05.728936 (+ 1080us) server_negotiation.cc:407] Connection header received
0929 19:05:05.729318 (+ 382us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:05:05.729319 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:05:05.729329 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:05:05.729358 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:05:05.729689 (+ 331us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53074: BlockingRecv error: recv got EOF from 127.0.0.1:53074 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260929 19:05:05.730366 23056 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:05:05.727603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36228 (local address 127.0.98.62:42891)
0929 19:05:05.727649 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0929 19:05:05.727651 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:05:05.728560 (+ 909us) server_negotiation.cc:407] Connection header received
0929 19:05:05.728683 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:05:05.728684 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:05:05.728696 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:05:05.728728 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:05:05.730305 (+ 1577us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36228: BlockingRecv error: recv got EOF from 127.0.0.1:36228 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260929 19:05:05.731346 23132 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:05:05.728199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58624 (local address 127.0.98.61:36095)
0929 19:05:05.728239 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0929 19:05:05.728240 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 19:05:05.730777 (+ 2537us) server_negotiation.cc:407] Connection header received
0929 19:05:05.730884 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:05:05.730885 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:05:05.730895 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:05:05.730924 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:05:05.731287 (+ 363us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58624: BlockingRecv error: recv got EOF from 127.0.0.1:58624 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260929 19:05:05.733798 23132 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:05:05.733048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58638 (local address 127.0.98.61:36095)
0929 19:05:05.733082 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0929 19:05:05.733084 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:05:05.733283 (+ 199us) server_negotiation.cc:407] Connection header received
0929 19:05:05.733394 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:05:05.733395 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:05:05.733413 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:05:05.733442 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:05:05.733753 (+ 311us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58638: BlockingRecv error: recv got EOF from 127.0.0.1:58638 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260929 19:05:05.736795 23132 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:05:05.736067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58646 (local address 127.0.98.61:36095)
0929 19:05:05.736100 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0929 19:05:05.736102 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:05:05.736317 (+ 215us) server_negotiation.cc:407] Connection header received
0929 19:05:05.736408 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:05:05.736409 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:05:05.736419 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:05:05.736448 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:05:05.736751 (+ 303us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58646: BlockingRecv error: recv got EOF from 127.0.0.1:58646 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260929 19:05:05.740765 23132 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:05:05.740037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58656 (local address 127.0.98.61:36095)
0929 19:05:05.740069 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0929 19:05:05.740071 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:05:05.740281 (+ 210us) server_negotiation.cc:407] Connection header received
0929 19:05:05.740372 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:05:05.740372 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:05:05.740382 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:05:05.740411 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:05:05.740720 (+ 309us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58656: BlockingRecv error: recv got EOF from 127.0.0.1:58656 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260929 19:05:05.746773 23132 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:05:05.745999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58662 (local address 127.0.98.61:36095)
0929 19:05:05.746046 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0929 19:05:05.746048 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:05:05.746278 (+ 230us) server_negotiation.cc:407] Connection header received
0929 19:05:05.746373 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:05:05.746374 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:05:05.746383 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:05:05.746409 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:05:05.746726 (+ 317us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58662: BlockingRecv error: recv got EOF from 127.0.0.1:58662 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260929 19:05:05.757050 23132 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:05:05.756127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58670 (local address 127.0.98.61:36095)
0929 19:05:05.756167 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0929 19:05:05.756169 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:05:05.756438 (+ 269us) server_negotiation.cc:407] Connection header received
0929 19:05:05.756542 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:05:05.756543 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:05:05.756554 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:05:05.756595 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:05:05.756994 (+ 399us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58670: BlockingRecv error: recv got EOF from 127.0.0.1:58670 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260929 19:05:05.776576 23132 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:05:05.775605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58680 (local address 127.0.98.61:36095)
0929 19:05:05.775651 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0929 19:05:05.775653 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:05:05.775959 (+ 306us) server_negotiation.cc:407] Connection header received
0929 19:05:05.776070 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:05:05.776072 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:05:05.776085 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:05:05.776115 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:05:05.776506 (+ 391us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58680: BlockingRecv error: recv got EOF from 127.0.0.1:58680 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260929 19:05:05.810999 23132 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:05:05.810020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58696 (local address 127.0.98.61:36095)
0929 19:05:05.810063 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0929 19:05:05.810066 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 19:05:05.810350 (+ 284us) server_negotiation.cc:407] Connection header received
0929 19:05:05.810451 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:05:05.810452 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:05:05.810463 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:05:05.810493 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:05:05.810920 (+ 427us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58696: BlockingRecv error: recv got EOF from 127.0.0.1:58696 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260929 19:05:05.877256 23132 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:05:05.876493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58710 (local address 127.0.98.61:36095)
0929 19:05:05.876581 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0929 19:05:05.876583 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:05:05.876705 (+ 122us) server_negotiation.cc:407] Connection header received
0929 19:05:05.876769 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:05:05.876770 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:05:05.876780 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:05:05.876845 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:05:05.877196 (+ 351us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58710: BlockingRecv error: recv got EOF from 127.0.0.1:58710 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260929 19:05:06.007423 23132 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:05:06.006437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58720 (local address 127.0.98.61:36095)
0929 19:05:06.006655 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0929 19:05:06.006657 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:05:06.006757 (+ 100us) server_negotiation.cc:407] Connection header received
0929 19:05:06.006858 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:05:06.006859 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:05:06.006872 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:05:06.006959 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:05:06.007324 (+ 365us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58720: BlockingRecv error: recv got EOF from 127.0.0.1:58720 (error 108)
Metrics: {"server-negotiator.queue_time_us":186}
W20260929 19:05:06.265254 23132 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 19:05:06.264588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58724 (local address 127.0.98.61:36095)
0929 19:05:06.264659 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0929 19:05:06.264661 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:05:06.264738 (+ 77us) server_negotiation.cc:407] Connection header received
0929 19:05:06.264828 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:05:06.264829 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:05:06.264840 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:05:06.264891 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0929 19:05:06.265207 (+ 316us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58724: BlockingRecv error: recv got EOF from 127.0.0.1:58724 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
I20260929 19:05:06.634549 23235 catalog_manager.cc:1259] Loaded cluster ID: b4cd01921cdb449bbc1fa28b2543fabd
I20260929 19:05:06.634626 23235 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 01cdb18cc48a4c0ab9d8b982248deef6: loading cluster ID for follower catalog manager: success
I20260929 19:05:06.635452 23235 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 01cdb18cc48a4c0ab9d8b982248deef6: acquiring CA information for follower catalog manager: success
I20260929 19:05:06.635773 23235 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 01cdb18cc48a4c0ab9d8b982248deef6: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 19:05:06.685366 23264 catalog_manager.cc:1259] Loaded cluster ID: b4cd01921cdb449bbc1fa28b2543fabd
I20260929 19:05:06.685446 23264 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 06a3d3e221a14b47b1f537f4dd6e1bf7: loading cluster ID for follower catalog manager: success
I20260929 19:05:06.686270 23264 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 06a3d3e221a14b47b1f537f4dd6e1bf7: acquiring CA information for follower catalog manager: success
I20260929 19:05:06.686607 23264 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 06a3d3e221a14b47b1f537f4dd6e1bf7: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 19:05:06.781240 23337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:06.778456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58740 (local address 127.0.98.61:36095)
0929 19:05:06.778598 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0929 19:05:06.778600 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:05:06.778655 (+ 55us) server_negotiation.cc:407] Connection header received
0929 19:05:06.778764 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:05:06.778765 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:05:06.778814 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:05:06.778897 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 19:05:06.779128 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:06.779623 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:06.780495 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:06.780690 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:06.780750 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:05:06.780904 (+ 154us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 19:05:06.781012 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 19:05:06.781074 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0929 19:05:06.781107 (+ 33us) server_negotiation.cc:299] Negotiation successful
0929 19:05:06.781123 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":80,"threads_started":1}
I20260929 19:05:06.785343 23343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:06.783056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53076 (local address 127.0.98.60:42913)
0929 19:05:06.783168 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0929 19:05:06.783169 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0929 19:05:06.783178 (+ 9us) server_negotiation.cc:407] Connection header received
0929 19:05:06.783224 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:05:06.783224 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:05:06.783259 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:05:06.783329 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 19:05:06.783431 (+ 102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:06.783908 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:06.784628 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:06.784831 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:06.784878 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:05:06.785016 (+ 138us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 19:05:06.785115 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 19:05:06.785172 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0929 19:05:06.785209 (+ 37us) server_negotiation.cc:299] Negotiation successful
0929 19:05:06.785225 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90,"thread_start_us":68,"threads_started":1}
I20260929 19:05:06.790102 23343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:06.787074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53078 (local address 127.0.98.60:42913)
0929 19:05:06.787126 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0929 19:05:06.787128 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:05:06.787281 (+ 153us) server_negotiation.cc:407] Connection header received
0929 19:05:06.787309 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:05:06.787310 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:05:06.787322 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:05:06.787429 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 19:05:06.787648 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:06.788482 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:06.789387 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:06.789670 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:06.789739 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:05:06.789859 (+ 120us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 19:05:06.789887 (+ 28us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 19:05:06.789932 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0929 19:05:06.789986 (+ 54us) server_negotiation.cc:299] Negotiation successful
0929 19:05:06.790004 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260929 19:05:06.790508 23349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:06.786994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36238 (local address 127.0.98.62:42891)
0929 19:05:06.787166 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0929 19:05:06.787168 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0929 19:05:06.787178 (+ 10us) server_negotiation.cc:407] Connection header received
0929 19:05:06.787224 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:05:06.787225 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:05:06.787281 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:05:06.787409 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 19:05:06.787573 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:06.788140 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:06.789095 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:06.789450 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:06.789536 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:05:06.789708 (+ 172us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 19:05:06.789823 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 19:05:06.790078 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0929 19:05:06.790097 (+ 19us) server_negotiation.cc:299] Negotiation successful
0929 19:05:06.790133 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":59,"threads_started":1}
I20260929 19:05:06.790630 23337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 19:05:06.787207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58746 (local address 127.0.98.61:36095)
0929 19:05:06.787258 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0929 19:05:06.787261 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 19:05:06.787389 (+ 128us) server_negotiation.cc:407] Connection header received
0929 19:05:06.787522 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 19:05:06.787523 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 19:05:06.787541 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 19:05:06.787605 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 19:05:06.787733 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:06.788549 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:06.789703 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 19:05:06.790022 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 19:05:06.790065 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 19:05:06.790283 (+ 218us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 19:05:06.790314 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 19:05:06.790365 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0929 19:05:06.790493 (+ 128us) server_negotiation.cc:299] Negotiation successful
0929 19:05:06.790517 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260929 19:05:06.792222 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUbnLHE/build/release/bin/kudu with pid 22985
I20260929 19:05:06.797348 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUbnLHE/build/release/bin/kudu with pid 23060
I20260929 19:05:06.802199 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUbnLHE/build/release/bin/kudu with pid 23135
2026-09-29T19:05:06Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2846 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5702 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (259003 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST